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Abstract 

We prove a refinement of the global Gan-Gross-Prasad conjecture essentially pro- 
posed by Ichino-Ikeda and N. Harris for unitary groups under some local conditions. 
We need to assume some expected properties of L-packets and some part of the local 
Gan-Gross-Prasad conjecture. 
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1 Introduction 

In this article, as a sequel of [50], we prove a conjectural refinement of the global Gan- 
Gross-Prasad conjecture ([(>]) for unitary groups under some local conditions. In a influential 
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paper [23], Ichino and Ikeda first formulated tlie refinement for ortliogonal groups. After 
the Ichino-Ikeda formulation, N. Harris considered the case of unitary groups in his Ph.D 
Thesis at UCSD ([21]). Our version of the refinement is slightly different from theirs at the 
archimedean place. 



The conjecture of Ichino— Ikeda and N. Harris. We now recall the conjectural refine- 
ment. Let E/F he a. quadratic extension of number fields with adeles denoted by A = and 
A^; respectively. Let y be a Hermitian space of dimension n + 1 and W a (non-degenerate) 
subspace of codimension one. Denote the unitary groups by U{V) and U{W) respectively. Let 
G = U{W) X U(y) be the product and H the diagonal embedding of U{W) into G. Denote 
G' = GLn,E X GLn+i,E as an algebraic group over E. Let tt = 7r„ g) 7r.„+i be a cuspidal auto- 
morphic representation of G{A) and let TCi^E be the base change of tTj to GLj(A£;), i = n,n+l. 
Denote by L(s, tte) the Rankin-Selberg convolution L-function L(s, 7rn,E x ''^n+i,E)- It is known 
to be the same as the one defined by the Langlands-Shahidi method. The reader may consult 
the introduction of [10] for an overview of the study of the central value i^(l/2, tt^^e x 7rn+i,E)- 
We also consider the adjoint L-function of vr 

L(s, 71, Ad) = L{s, TTn, Ad)L(s, 7r„+i, Ad). 

Denote the constant 

n+1 

A„+i = Yl Lih v') = ^(1, ^)^(2, lE)Li3, r])---L{n + l,r] 



n+l^ 



where t] is the quadratic character of A^ associated to E/F by class field theory. Note 
that here A^+i = L(M^(1)) where is the motive dual to the motive M associated to 
the reductive group U{n) defined by Gross ([13]). We will be interested in the following 
combination of L-functions 

(1.1) ^(.,vr)^A„,i- ^^"'^^^ 



L{s + l,7r, Arf)' 

We also write =Sf (s, tt^,) the local component at v. 

Let [H] denote the quotient H{F)\H{A) and similarly for G. We endow H{A) (G'(A), 
resp.) with their Tamagawa measures ^ and [H] {[G], resp.) with the quotient measure by 
the counting measure on H{F) {G{F), resp.), cf. §2. In [6], Gan, Gross and Prasad propose 
to study an automorphic period integral 

^H{<f)) := f mdh, <j) s IT. 

J[H] 



^ Since the unitary group H has nontrivial central torus, we need to introduce a convergence factor: dh 
1/(1, r])~^ Yiv ^(1: ^u)|w|t, for a nonzero invariant differential uj on H . Similarly for G. 
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They conjecture that the non-vanishing of the hnear functional on vr (possibly by switching 
to another member in the Vogan L-packet of n) is equivalent to the non- vanishing of the central 
value L(|,7r£;) of the Rankin-Selberg L-function. This conjectural equivalence is proved for tt 
satisfying some local conditions in our previous paper ([50]). One direction of the equivalence 
was also proved by Ginzburg-Jiang-Rallis (cf. [10], [11]). 

For arithmetic application, it is necessary to have more precise relation between the au- 
tomorphic period integral ^ and the central L-value. To state the precise refinement of 
the Gan-Gross-Prasad conjecture, we need to introduce more notation. Let (■,-)pet be the 
Peterson inner product 



((j),(p)pet= \ (t){g)ip{g)dg, 0,(/?e7r. 



'[G] 

Fix an decomposation as a product 

V 

under the decomposition vr = (x)7rt,. In this way we fix an invariant inner product on 7r„. 
Ichino and Ikeda first consider the following integration of matrix coefficient: for <pv,fv ^ t^v, 
we define 

It has the following nice properties: 

1. If Ti^ is tempered, it converges absolutely and a{(f)y, (/)y) ^ 0. 

2. When tt^ is unramified^, and the vectors (p, if are fixed by such that ipv)v = 1, 
we have ^ 

3. If Homi^(i?^)(7r„, C) 7^ 0, then the form does not vanish identically. 

The first two are proved by Ichino and Ikeda (N. Harris in the unitary case). The third 
property is conjectured by them and proved by Sakellaridis and Venkatesh ([31]) in more 
general setting. Waldspurger also proved the third property in the orthogonal case. Because 
of the second property, we normalize the form a„ 



\{(t)v,^v) = rf (7rv{h)(j)y,(fy)ydh. 



^For a place v we say that 7r„ is unramificd if the quadratic extension E/F is unramified at v, the group 
G{Fy) has a hyperspecial subgroup Ki, = G{Oy) and tt^ has a nonzero i^T^-fixed vector. 



3 



We are now ready to state the conjecture of Ichino-Ikeda and N. Harris (cf. [23], [21]) that 
refines the global Gan-Gross-Prasad conjecture for unitary groups. Assume that the measure 
on H{A) and the measures on H{Fy) defining satisfy: 



Conjecture 1.1. Assume that tt is tempered. For any decomposable vector cf) = (x)(/)^, g tt = 
®7ry, we have 

(1.2) I'^wi' ^ tt) n 

where S^^ is a finite elementary 2-group: the component group associated to the L-parameter 
o/tt = 7r„ g)7r„+i. 

The conjectural formula of this kind goes back to the celebrated work of Waldspurger ([3S]) 
for the central values of L-functions of GL(2) (more or less equivalent to the case U{1) x U{2) 
in the unitary setting). An arithmetic geometric version generalizing the formula of Gross- 
Zagier and S. Zhang ([16], [43]) is also formulated in [48]. More explicit formulae were obtained 
by Gross ([ ! '-]), S. Zhang ([47]) and many others. The formula of Waldspurger and the formula 
of Gross-Zagier and S. Zhang ([16], [45], [46], [43]) play an important role in the spectacular 
development in application to the Birch and Swinnerton-Dyer conjecture for elliptic curves 
in the past thirty years. More recently, Y. Tian ([37]) applies both formulae together to a 
classical Diophantine question and proves the infinitudes of square-free congruent numbers 
with an arbitrary number of primes factors. 

The refined global conjecture for SO (3) x SO (4), concerning "the triple product L- function" , 
was stablished after the work by Garret [8], Piateski-Shapiro-Rallis, Harris-Kudla [19], Gross- 
Kudla, and Ichino [22]. Recently Gan and Ichino ([7]) established some new cases for 
SO (4) X SO (5) (for endoscopic L-packets on SO (5)). All of the known cases utilize the theta 
correspondence in an ingenious way. 

The formula of Waldspurger was also reproved by Jacquet and Jacquet-Chen ([3]) using 
relative trace formulae. Our method is more close to theirs. 

Main results. We now state our main result. Throughout this paper, we assume the 
expected properties the L-packets of unitary groups (for all Hermitian spaces W, V ), analogous 
to the work of Arthur on orthogonal groups (cf. [31]. [11] for the progress towards this end). 
Moreover we need to assume a part of the local Gan-Gross-Prasad conjecture in the unitary 
case for all local fields of characteristic zero: in a tempered (Vogan) L-packet there is at 
most one member that admits a nonzero invariant linear form. A proof of this part (in fact 
an even stronger result) for p-adic fields is recently posted by Beuzart-Plessis ([2, Theorem 
1]); the local conjecture in the orthogonal case for p-adic fields has earlier been proved by 
Waldspurger. 
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We also assume that the fundamental lemma of Jacquet-Rallis holds for all p-adic field. 
When n = 2 and p > 2, this was verified by the author by brutal computation. In [11] and its 
appendix by J. Gordon, the fundamental lemma is proved when p is larger than a constant 
c{n). It is plausible that one may deduce the general case from their results. Without 
assuming this, in the theorem below we will need to impose one more condition on the 
quadratic extension E/F: it is split for all non-archimedean place with residue characteristic 
^ c{n). 

At a split place v, we may also define an alternative normalization via the Whittaker 
model: 

See §3 for the precise definition of A„ in terms of the Whittaker model and normalization of 
the inner product on the Whittaker model. When v is non-archimedean, we will show that 

It is expected to be true for archimedean places, too. Also we set a'^ = ay for non-spht v. We 
again normalize 

a'X(l^v,^v) = -^Tj 7ay{(f)y,^y). 

Theorem 1.2. Let tt he tempered cuspidal automorphic representation of G{A). Denote S 
by the set of non-split places v of F where Wy is not unramified. Assume that 

(i) The local components tt^ are supercuspidal for at least one split place. 

(a) If V e H, then either Hy is compact or TTy is supercuspidal. 

Then we have results for the following two (extreme) cases according to G{M.): 

(1) The totally split case: if every archimedean place v of F is split in the extension E/F 
(i.e., Gpao — (GL„ X GLn+i)Fao)> then we have 

((j),(j))pet V2' ((pv,<Pv}v ' 



(2) The totally definite case: if G{F^) is compact where F^ = 0?;|oc -^'^ particular, E is 
a CM extension of a totally real field F ), then the same equality holds up to a non-zero 
constant c^^ depending only on the archimedean component tTqo of ti: 
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Under our assumptions (i), the automorphic representation vr has cuspidal base change to 
the general linear group and hence 

The condition (z) is due to the fact that currently we do not have a complete spectral 
decomposition of the relative trace formulae of Jacquet and Rallis. The condition (ii) seems 
to be only a technical restriction for our approach and will be discussed in §9. The restriction 
for the archimedean place is due to the lack of the existence of smooth transfer at archimedean 
places (cf. §5). However, even if we assume the existence of smooth transfer, it is probably 
substantially more technical to remove the restriction in the archimedean case (say, to evaluate 
the constant Cna,). 

We also make a local conjecture (Conjecture 4.3) for each place v. Assuming this conjec- 
ture, we may improve Theorem 1.2 to only assume that the base change he is cuspidal. 

Remark 1. At a split place, in practical computation it seems easier to use the normal- 
ization via Whittaker model instead of the matrix coefficient integration, for example, the 
archimedean computation by Watson for the triple product L-function ([42]). 

Remark 2. In our previous paper [50], we proved the equivalence of non- vanishing of the two 
sides of the equality (in more generality). It is worth mentioning that the current result relies 
on this equivalence; indeed the equivalence shows that if LHS is zero, so is the RHS. What 
the current paper actually deals with is the case when the LHS is non- zero. 

Remark 3. Note that for a non-archimedean place v, is possibly compact only when n < 2. 
When n = 1, Hy is always compact for a non-split v. This compact case already occurred in 
[3]. 

Some applications. We have the following application to the positivity of some L-values. 
The positivity is also predicted by the grand Riemann hypothesis. Lapid has obtained a more 
general result for Rankin-Selberg central L-values via a different method ([30], cf. also [29] 
for the positivity of central values of L-functions of symplectic type). 

Theorem 1.3. Assume that tt is as above and E/F is split at all archimedean places. Then 
we have 

Proof. It suffices to show this when L^tie, |) ¥= 0. Then by [50], there exists vr' in the same 
L-packet of tc such that the period ^ on n' does not vanish. By replacing vr by vr', we may 
assume that the space Hom^(v(/)(i?„)(vr„, C) does not vanish for every v. Then the local terms 
a'jj do not vanish. Now the positivity follows from the fact that the a'^ are all positive definite, 
and the other L-values appearing in ^(tt^;, 1/2) except L{7Ve, |) are all positive. □ 

Remark 4. As another application, M. Harris showed that the Ichino-Ikeda conjecture would 
imply the algebraicity of the L- value =2'(l/2,7r) up to some simple constant when G{Fcc) is 
compact and aoc (cf. [20, §4.1]). 
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Outline of proof. We now sketch the main ideas of the proof. First of all, by the multi- 
plicity one result ([1],[36]), we know a prior that there is a constant denoted by depending 
on TT such that for all decomposable 0, e tt: 

(1.3) ^{ct^)¥{^ = n al{(t>., ^v). 

V 

Instead of working with individual (j) as in the conjecture, we switch our angle of view to 
working with a distribution attached to tt: 

(1.4) Uf) := ^{<mWW), f e K^^{G{A)), 

<t> 

where the sum of (p is over an orthonormal basis of tt. In this article, we call this distribution 
a (global) spherical character following the names of many of its early analogues (cf. [32] etc.). 
We may accordingly define a local spherical character. 

(1-5) Jl{f,):=Y,c^lMfvHAv), /.e^»(G(F,)), 

<t>v 

where the sum of 0„ is over an orthonormal basis of tt^,. By (1.3), we clearly have for decom- 
posable f = (S)yfv- 

(1-6) J.(/)='Cn"^-(/'')' 

V 

where in the product of RHS, for a given n and /, the local term J^^(fy) = 1 for all but 
finitely many v. Then we have the following consequence of Conjecture 1.1. 

Conjecture 1.4. For all f = 0^ fv ^ ^^°°(G(A)) , we have 

Mf) = n^,i/2)YlJliA,). 

V 

In fact. Conjecture 1.4 also implies, hence is equivalent to Conjecture 1.1. To see this, we 
note that the following are equivalent: (1) Hom(7r^,C) 7^ 0, (2) ay 7^ 0, (3) the distribution 
J'l^ does not vanish. ^ Hence, Conjecture 1.1 holds if for some v the linear form vanishes. 
Now assume that for all the linear forms do not vanish. Then the distributions Jjr^, do not 
vanish. Then by Conjecture 1.4, the constant 'iCr must be ^{n, 1/2) which implies Conjecture 
1.1. Similarly, Theorem 1.2 follows from similar conjecture as 1.4 where one replaces by 
a'^ when defining J^^. 

Moreover, there is a parallel question on general linear groups and it is essentially reduced 
to the celebrated theory of "Rankin-Selberg convolution" ([25]). Therefore, a natural idea 



■^It is clear that (2) is equivalent to (3). The equivalence of (1) and (2) follows from the third property of 
av listed earlier. 
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is to transfer the question from unitary groups to general linear groups via (quadratic) base 
change from unitary groups to general linear groups. We are therefore lead to comparing the 
two period integrals via the relative trace formulae of Jacquet and Rallis ([26]). 

In the general linear case, there is a decomposition of a global spherical character into 
local ones, analogous to Conjecture 1.4. But this time one may prove it without difficulty 
(indeed almost tautological). Hence, to deduce the Conjecture 1.4, it suffices to compare 
the two local spherical characters. Moreover, since we only need to find the constant ^7^? 
we may just choose some special test functions / (as long as the RHS does not vanish for 
our choice). Therefore the main innovation of this paper is a formula for the local spherical 
character evaluated at some special test functions. The formula can be viewed as a truncated 
local expansion of the local spherical character that is analogous to the local expansion of 
characters due to Harish-Chandra. The result may be of independent interest in view of local 
harmonic analysis in the relative setting. 

For comparison, let us recall briefly the result of Harish-Chandra. Let F be a p-adic flelds, 
G the F-points of a connected reductive group, and q the Lie algebra of G. Let M be the 
nilpotent cone of q and denote by JV/G the set of G-conjugacy classes in A/", which is a flnite 
set. Let /io be the nilpotent orbital integral associated to O e M jG. The exponential map 
deflnes an homeomorphism exp : w ^ f2 where u (f2, resp.) is some neighborhood of in g (1 
in G, resp.). Let vr be an irreducible admissible representation of G. Then Harish-Chandra 
showed that there are constants co(7r) indexed by C e M jG such that when f2 is sufficiently 
small, for all / supported in fi: 

(1-7) tr(vr(/))= 2 Co(7r)/io(7^). 

Here is the function on a; via the homeomorphism, and /[, its Fourier transform. The 
constants Cc)(vr) contains important information of vr. For example, there is a distinguished 
nilpotent conjugacy class, namely the conjugacy of g M . If vr is a discrete series representa- 
tion, the constant C{o} (tt) is essentially the formal degree of vr provided an appropriate choice 
of Haar measure. 

Now we return to our local setting for a non-split non-archimedean place v of F . We 
suppress the subscript v. We consider the local spherical character in the general linear case. 
Let G' := Res^;/i?(GL„ x GL„+i) and let H be irreducible unitary generic representations of 
G'{F). Then the local spherical character /n (cf. (3.17)) defines a distribution on G'{F) with 
certain invariance property. These distributions are related to distributions on the F-vector 
space: 

s„,+i = {XeM„+i(F)|X+X=0}. 

The group GLn+i{F) acts on s„+i by conjugation and we will be interested in the restriction 
of the action to the subgroup GLn{F) (as a factor of the Levi of the parabolic of (n, l)-type). 
We let w be a small neighborhood of in the F-vector space Sn+i- Then we have a natural 
way to pull back a function /' on a small neighborhood of 1 in G' to a function denoted by 
on uj (cf. §8 for the precise definition). It is tempting to guess that there exists an analogous 
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expansion of In in terms of the (relative to GI/„(F)-action) unipotent orbital integrals on 
Sn+i- ^ However, so far there are some difficulty. For example, when n ^ 2 there are infinitely 
many GL„(F)-nilpotent orbits in Sn+i and these nilpotent orbital integrals often need to be 
regularized. We then define a subspace of admissible functions on u. The precise definition is 
very technical. We expect that admissible functions have vanishing nilpotent orbital integrals 
(however, generally not even defined so far), except for one of the two regular unipotent 
orbits denoted by An expansion such as (1.7) of /n(/) would then tell us that there 
should be only one term left, corresponding to the regular unipotent orbit Though it 
seems challenging to prove something such as (1.7) in our setting, we nevertheless manage to 
establish a truncated version (see Theorem 8.4 for the detail): 

Theorem 1.5. Let U be irreducible unitary generic representations of G'{F). Then there is 
a sufficiently small oj such that for every admissible functionf, 

/n(/') = (*K-(A')^0, 

where (*) is an explicit non-zero constant independent ofU (up to its central character). 

We have a similar result for a local spherical character J^r on the unitary group when either 
TT is a supercuspidal representation or the group U{W) is compact. See §9 for more details 
(Theorem 9.7). Then our main result follows from the local comparison of the two spherical 
characters (cf. §4 Conjecture 4.3). 

The proof for the unitary case seems to be harder and needs the full strength of our 
previous results in the companion paper [50]. Namely we have to make use of the following 
results (cf. §9) 

1. The existence of smooth transfer. 

2. Compatibility of smooth transfer with Fourier transform. 

3. Local (relative) trace formula on "Lie algebra" . 

Note that the proof in [50] of the these ingredients are in the reverse order listed here. 

Structure of this paper. After fixing some notations in §2, we review several global 
periods involving general linear groups in §3 and deduce the decomposition analogous to 
Conjecture 1.4. Then in §4 we recall the RTF of Jacquet and Rallis and reduce the question 
to a comparison of local spherical characters. Then we give the proof of Theorem 1.2 assuming 
a local result. In §5 we deal with the totally definite case (i.e., G(M) compact). In §6 we 
prepare some (relative) harmonic analysis on Lie algebras. In §7 and §8, we prove the local 
character expansion for the general linear group. In §9 we show the local character expansion 
for the unitary group under some conditions, and complete the proof via the comparison of 
both characters. 

''Relative to Gi„(i^)-action, an X e s„+i is "nilpotent" if the closure of its GLn (F)-orbit contains zero. 
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Finally, we warn the reader of the change of measures: Only in the introduction, we use 
the Tamagawa measures associated to a differential form u on H normalized by: 

dh = L{l,r])-'YlL{l,7],)\ul. 

V 

To have a natural local decomposition, below we will immediately switch to 

(i/i = ]^ (i/it,, = ]^ L(l, ?7„)|a;|^,. 

V V 

Another change comes when we go to the local setting where we even consider the unnormal- 
ized local measure 

dhy = Iwli). 

Similar warning applies to other groups such as G and the general linear groups. 
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2 Measures 

Measures related to general linear groups. We first list the main notations and con- 
ventions throughout this paper. We denote Hn = GL„, its standard Borel i?„ with A„ the 
diagonal torus, the unipotent radical. We denote by Nn- the opposite unipotent subgroup 
of Hn, and an open subvariety H'^ = N^AnNn- of if„ (essentially the open cell of Bruhat de- 
composition). Their Lie algebras are denoted by f)„, n„ etc. We denote Mn^rn{F) the F- vector 
space of all n x m-matrices with coefficients in F\ and if rz, = m we write it as Mn{F). Then 
it comes with the natural embedding Hn c Mn,n- We denote e„ = (0, 0, 0, 1) g Mi^„(F), 
e* e Mn,i{F) the transpose of e„. 

We usuaUy understand Hn^i as a subgroup of Hn via the block-diagonal embedding 

Hn-i B h^-> ^ 1 ) ^ 

We thus have embeddings ... c Hn-2 ^ -f^n-i ^ Hn- Similarly A'"„_i as a subgroup of etc.. 

For a quadratic extension E/F, we assume that F = F[t] for t = VS. We write E^ the 
F- vector space where the nontrivial Galois automorphism in Gal(i?/F) acts by +1. Locally 
we will fix an additive character i/j = ipp oi F and then define a character ipE oi E hy 
i^E^z) = ij}{\irE/Fz) for the trace map tiE/F : E ^ F. We also say that ■0 is unramified 
if the largest ideal of F over which ■0 is trivial is Op- Similarly for ipE- We will consider 
the self-dual measure on F, E~ and E. Note that if we use the isomorphism F ^ E^ by 
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X t-^ V^x, then the measure on E is for the self-dual measure dx on E . Here our 

absolute values on F and E are normalized such that 

d{ax) = \a\FX, x s F 

and similarly for E. On Mn{E) there is a bilinear pairing given by tT{XY) valued in E. We 
then have a Fourier transform for e {Mn{E)) 



4>{X) := f 4>{Y)ijE{i^{XY))dY, 



for the self-dual measure on Mn{E). Note that this is also the same measure by identifying 
Mn{E) with E^ and using the self-dual measure on E. 

On F^ we denote the normalized Tamagawa measure associated to the differential form 
x~^dx: 

d-X = CF{lh'^^ 



\X\f 



and the unnormalized one: 

d*x 



\x\f 

Similarly for E^ . On Hn{F) we will take the Haar measure 



and similarly for Hn{E) (where we replace C-f(I) by Ce{^))- Sometimes we also shorten | det(g')| 
by \g\ if no confusion arises. 
For X s Mn^m{F), we define 

||x|| = max{\xij\F}- 

We will assign the measure on Nn{F) the additive self-dual measure via the identification 
Nn{F) with the F- vector space Similarly we normalize the Haar measure on A„(F) 

by identifying it with (F^)". 

We denote the modular character by 



5„(a) = det{Ad{a) : Un) = Y\ 



n 



\a. 



■i\F 



i=l 



where a = rfzaf^fai, a2, aj e An{F) acts on n by Ad{a)X = aXa~^ with unitary group 
U{y). Similarly, we have 5n,E if we replace F by E. 

Now if F is a number field, we endow the group Hn,{A) with the product measure 

dg = Y\d9v 

V 

Similarly for i/„(A£;) if F is a quadratic extension. 
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Measures related to unitary groups. Our method involves comparison of orbital inte- 
grals between the unitary and general linear cases, and between their Lie algebras, respectively. 
We thus need to choose compatible measures on them. Let ^ be a Hermitian matrix of size 
n + 1. Then we may and will view the group U{6){F) as the subgroup of GLn.+i{E) consisting 
of g s GL„+i(£^) such that 

f ■ Oge~' = 1. 

We may and will view the Lie algebra u{9) of U{9) as the subspace of Mn+i{E) consisting of 
X G Mn+i{E) such that 

X* + 6X0-^ = 0. 

We denote by u(6')^ a companion space where the last equality is replaces by 

X* = exe-\ 

For any number t e E such that r = — r ¥^ 0, we have an isomorphism from u(^) to u(6')^ 
mapping X to t^^X. 

We will need to consider the symmetric space Sn+i{F) ^ Hn+i{F)\Hn+i{E) , and identify 
it with the subspace of GL„+i(£') consisting of e GL„+i(i?) such that 

99 = 1- 

We also have the tangent space s = Sn+i of 1 e Sn+i, which we call the Lie algebra of Sn+i{F) . 
Wiewed as a subspace of M„+i(E'), the vector space s consists of X e Mn+i{E) such that 

X + X = 0. 

Its analogous companion is the space M„+i(F) (or gln+i{F)) viewed as a subspace of Mn+i{E), 
namely consisting of X e Mn+i{E) such that 

X = X. 

For any number t e E such that r = — r 7^ 0, we have an isomorphism from s{F) to M„+i(F) 
mapping X to t~^X. 

We have a canonical bi-£^-linear pairing given by (x, y) 1^ ti^xy) e E. We consider both 
s and u as F- vector subspaces of Mn+i{E). The restrictions of the bilinear form on Mn+i{E) 
to s and u take values in F and are non-degenerate. Moreover, if we change the Hermitian 
matrix defining u to an equivalent one, the subspace u changes to its conjugate by an element 
in GL„+i(i?). The additive character i/j and i/^e determine a self-dual measure on M„+i(A£;), 
u(A), s(A) and the local analogues. These measures can also be treated as the Tamagawa 
measure associated to differential forms. Let Uq be the differential form on u that defines the 
self-dual measure. We use it to normalize the differential form that defines the measure on 
U{9){Fy). We consider the Cayley map 

c{X) := {1 + X){1 - X)-' 
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defines a birational map between u and U{9), and it is defined aX X = 0. We normalize the 
invariant differential form u on U (9) by requiring that the pull back c*u evaluating at is the 
same as uq evaluating at 0. It follows that, when v is non-archimedean, the Cayley map takes 
the restriction of the self-dual measure to a small neighborhood of in u to the restriction of 
the Tamagawa measure |a;|t, to a small neighborhood of 1 in U{6){Fy). In this way we define 
the Tamagawa measure on U{9){A), globally and locally, as follows: 

dh = Y\L{^,i1v)\^^\v 

V 

Finally, we note that under the Tamagawa measure, the volume of [^7(1)] = U (1)(F)\[/(1)(A) 
is given by 

vol([[/(l)]) = 2L(l,r/). 

3 Explicit local factorization of some periods 

In this section we decompose several global linear forms on general linear groups into explicit 
products of local invariant linear forms. Nothing is original in this section but we need to 
determine all constants for the proof of the main results. 

Invariant inner product. Let 11 = n„ be a cuspidal automorphic representations of 
Hni^E)- We recall some basic facts on the Whittaker model of 11 = ®^n^. We extend 
the additive character i/je a character of Nn,{E) by 

n-l 

Then the Fourier coefficient of is defined as 

W^{g) = (t){ug)ipE{u)du. 

JN{E)\N{Ae) 

Then we have s ^°°(A'!„(A£;)\iJ„(A£), -i/^^;). The map realizes an equivariant 

embedding H ^'*(A^„(Ai5)\if„(Ai5), ^e) with image denoted by W{Tl,i)E)- For e H = 
®yjlw We assume that is decomposable 

= n^-(^-)' ^ '^"{Nn{E^)\H,,{E^),^l^E,n,) 

w 

where w runs over all places of E. 

We compare the unitary structure in the decomposition 

U^(g)W{U^,ijE,n,). 
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On n we have the Petersson inner product, for 0, cj)' e 11: 




Jz„{Ab)//„(£;)\//„(a_b) 
On W{Jlu>)^E,w) we have an invariant inner product defined by 



Jn„-i{E^)\H„-i{E^) 




h 



1 



) 



dh. 



When n^„ and 'iIje,w are unramified, the vectors W = W are fixed by Hn{OE,w) and normahzed 
by W{1) = 1, we have 



= L(i,n^ X n^„). 

This can be deduced from [27] Therefore we deine a normahzed invariant inner product 



Then the product "'^So converges and defines an invariant inner product on W(n,'0^). It 
is a natural question to compare it with the Petersson inner product. We now recall a result 
of Jacquet-Shalika. 

Proposition 3.1. We have the following decomposition of the Petersson inner product in 
terms of the local inner product t^J^ : 



where = ^^W^ and = ®t„W^. 

Proof. Up to a constant this is proved by [27, §4]. We thus recall their proof in order to 
determine this constant and this will also be used below to decompose the Flicker-Rallis 
period. We consider an Eisenstein series associated to a Schwartz function $ g ^^°°(A^). let 
e = (0,0,...,1) e i?" and P is the mirabolic subgroup of Hn consisting of matrices whose last 
row is e. Set 



L(i,n^ X n^) 



(3.1) 




V 




Consider the Epstein-Eisenstein series 



(3.2) 





-yeZP{E)\Hn{E) 



which is absolutely convergent when Re(s) > 1. Equivalently, we have 




14 



(Note: this corresponds to the case r/ = 1 in [27, §4].) It has meromorphic continuation to C 
and has a simple pole at s = 1 with residue ([27, Lemma 4.2]) 

n 

Note that the only non-explicit constant denoted by c in [27, Lemma 4.2] is the volume of 
E^\A^. Now consider the zeta integral 



J(s, = E{g,<^,s)(j){g)(l)'{g)dg. 

'Z{kE)H„{E)\H„{KE) 



One one hand, it has a pole at s = 1 with residue 

Y0\{E^\K\) 



$(o)<0, 



Pet- 



ri 

On the other hand, when Re(s) is large, it also equal to the following integral 
(3.3) vl/(s,$,iy^,Vr^,) = f ^eg)WMW;'{9)\^^^{9)Yd9 

J N„{AE)\H„{kE) 

where, by [27, Prop. 2.3] for unramified data of {^w^Ww,W^) and 'ijjE,w at a place nor- 
malized such that W{1) = W{1) = $^(0) = 1, we have 

^(s,$,iy^,H^:) = voi(i^jL(s,n„ X n^). 

(Note: for our measure on Nn{Eyj), we have vol(A'^„(£'^) n X.^,) = 1.) From this we may 
deduce that 

T-r ^(s,$^,W^,», W,^/,^) 

^(s, $, w^, w^>) = L{s, n X n) 1 1 :^ , 

where the local factors are entire functions of s and for almost all w they are equal to one. 
Moreover, all local factors converge absolutely in the half plane Re(s) > 1 — e for some e > 
([27]). From this we deduce that its residue at s = 1 is given by another formula 



Re..-,L(.,nxIl)n^(^'^-^^--^^- 

V L(i,n^xn^) 



From the two formulae of the residue, we first deduce that i?,«(VF<^,^, Wij^'^w) is iI„,(-E'w)-invariant 
and second that 

^(1, w^^^, w^'^j = w^'^j. 

We consider the action of Hn{E) on the row vector space E^ from right multiplication. 
Then the stabilizer of e is the mirabolic subgroup P„. Let Nn^i,+ {Eyj) be the unipotent 
part of Pn and A^„ i _(_E'^„) the transpose of A^„^i_+ (£"„,). We consider the open dense subset 
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Nn^i^+Hn-iNn^i-Zn = PnNn^i-Zn- We may decompose the measure on Hn (or more precisely 
its restriction to the open subset) 

dg = I det{h)\~^ dn+dhdri-d* a, 

where 

Note also that the embedding ^ P„ induces an isomorphism Nn\Pn — Nn-i\Hn-i. For 
an integrable function / on Nn\Hn, we may write 



f f{g)dg 

W ) 



f{hn^a) \ det(/i)| ^dh j dn-d*a 

We now apply this formula to the integral VF^^^, PF<^/,^). Clearly -i?^ is P{Eyj)- 

invariant. Therefore we may write 



J 



^{l,^y,,W^^^,W^,^^) = I ^{ean_)§^(Jl^{an_)Wy,,Il^{an_)W'J\a\^d*adn_. 

1,1, -{Ew) 

For X E E"", (with last entry nonzero), let n_(X) be the element in ZnNn,i,~{Ew) with last 
row equal to X. We consider 

r{x) := M'nUn-{x))w^,uUn^{x))w:^), 

whenever it is defined. A suitable substitution yields 

^{l,^^,W^,.n,W^',..) = \ <^u.{X)T{X)dX. 

Since by the other residue formula, we also know that this is equal to a constant multiple times 
$u;(0) times an invariant inner product on >V(n^,'0E,^), for all and W^,, W^. We deduce 
that T{X) is a constant function (whenever it is defined). Therefore T{X) = '(?,«(VI4;, W^) 
and "i?^ is if„(£'^)-invariant. Moreover we now have 

^(1, w^,^, w^,,^) = ^^{w^, w:,) \ ^x)dx = Mw^, w^;)C(o). 

This completes the proof. We also note that if we use the //„(i?u;)-invariance of 'dw, which 
can be proved independently, then the proposition can be deduced immediately from the 
two residue formulae. From the if„£'^-invariance of -^^ and the equation (3.3), we may 
immediately deduce that 'dw = L(l,n^ x n^„) for unramified data, as promised earlier. 

□ 
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For later use, as we will be in the case of a quadratic extension of number fields, we will 
now consider Hn^E as an algebraic group over the base field F. Therefore we rewrite the result 

as 

(3.4) <,,v.=^^^a#^n^i(--''<)' 

where = n«)|i,''^w (^^^ ^"^0) places w above v. 

Flicker-Rallis period. Now let E/F be a quadratic extension of number fields and 11 = n„ 
a cuspidal automorphic representation of Hn{AE). Assume that its central character uyi\ax = 
1. We would like to decompose the Flicker-Rallis period ([1],[9]) explicitly. It can be viewed as 
a twisted version of the Petersson inner product (it indeed gives the Petersson inner product 
if we allow E = F x F tohe split globally). Therefore it is natural that the method is similar 
as well. 

We first assume that n is odd. Then we have the global Flicker-Rallis period, an i7„(A)- 
invariant linear form on 11: 

(3.5) = := r (j){h)dh, 0en. 

J Zn{A)H„{F)\H„{A) 

The global period /3 is related to the Asai L-function L(s,n,As+) (for the definition of As-, 
cf. [6]). We set 

en = diag{T''-\T^-\...,l)e Hn{E) 

and 

€n-i = T ■ diagir''-^ r'^"^ 1) = re^-i e Hn-i{E). 

(Note: T G E~ .) Indeed we may choose any e„ = diag{ai, a„_i, a„) such that ai/ai+i s E~ 
for i = — 1 and a„ = 1. We again use the Whittaker model of 11. We will again 

consider ifn.s as an algebraic group over F. In particular, we consider 11^, as a representation 
of Hn{Ey) where Ey = E (i)p F^ is a. semisimple Fy algebra of rank two. For a place f of F, 
we define for W„ e W{Ily,'ijjE^) 

(3.6) MWy) = f wj '"-^^ ^ ] dh. 

Jn„-i{F^)\H„-i(F^) \ ^ J 

It depends on the choice of r = \/6. For unramified data with normalization VF^(e) = 1, we 
have 

Py{Wy) = VOl(K„(O^J)L(l,n„As+). 

We thus define a normalized linear form 
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Proposition 3.2. We have an explicit decomposition 

vol(Fx\Ai) 



where W = = ®^W„ g >V(n, i^E)- 

Proof. Let $ e iS(A"') and we consider the same kind of Eisenstein series E{g, $, s) as in the 
previous paragraph, but now on i7„(A). Then we define 

:= f E{g,<^,s)<P{g)dg. 

JZ(A)Hn{F)\Hn{A) 

We then have ([1, pp.303]) 

(3.9) J(5>,$) = vl>(s,M/<^,<l>), 

where 

J N„{A)\Hr,{A) 

(Note that we have different definition of the additive character i/je-) Indeed, we have 
/(s,$,0)= f ^eg)<P{g)\g\-^dg 

JPn{F)\H„{A) 



(^{eg) ( (j){ng)dn j \g\''dg. 

JP„(F)Nn(A)\Hn(A) \JNn(F)\N(n(A) ) 



IP„iF)N„iA)\H„iA) \J N„{F)\N i„{A) 

We have a Fourier expansion 

M = 2 W^i^g). 

'yeN„{E)\P„{E) 

Only those 7 such that il^ E{in'^~'^) = 1 for all n e Nn{A) contribute nontrivially. Therefore 
we may replace the sum by 7 e enPn{F): 

Hs, $, (^) = f Heg) I V W^ie^^g) ] \g\'dg 

JPn{F)N„{A)\H„{A) I 



^{eg)W^{eng)\g\'dg 

Jn„(a)\h„(a) 



IN„{A)\H„{A) 



(Note that vol(A^„(F)\A^„(A)) = 1.) We define for each place v of F, 

^(5,iy,,<l>,)= f W,{€nh)^,{eh)\h\'dh. 

J N„{F^)\Hr,{F^) 



For unramified data, we have 

^(s, w;, = voi(K„(Op^j)L(s, n,, As+). 

And we have ([9, pp.185]) 

^(1,PV,,$,) = /3,(W,)$,„(0). 

Alternatively we may prove this using (3.9), analogous to the proof of Prop. 3.1. Again, 
analogous to the proof of Prop. 3.1, we may take residue of (3.9) to obtain: 

^^^^t^^^S(O)/3(0) = Res.^Ms, n, A5+)8(0) U /35(iy„) 
n ^ ^ 

V 

This completes the proof. □ 
When n is even, we insert the character r] in the definition of /3 

(3.10) /3(0) = /3„((/*) := f (l){h)r,{h)dh, e n. 

JZ„{A)H„{F)\H„{A) 

The Asai L-function is then replaced by L(s, 11, As^), or we may write it as L(s, 11, As*^^^^" ^). 
We also modify the definition 

(3.11) /3.(W.) = f wJ '"-^^ ) riv{h)dh. 

J7V„_i(F,)\/f„_i(F,) V / 

The same argument shows that (3.8) still holds. 



Rankin-Selberg period. We now follow [25]. Let 11 = n„ (g) Iln+i and 11, a cuspidal 
automorphic representations of Hi{AE), i = n,n + 1. We define the global Rankin-Selberg 
period as 

(3.12) A(0) = r (j){h)dh, 0en, 

J H„{E)\H„{Ae) 

where i/„ embeds diagonally into Hn x Hn+i- To decompose it, we need the Whittaker 
model yV{Iln,4'E) (^O^n+i^i^E), resp.) of n„+i with respect to the additive character ifj^ 
{ipEi resp.). We define a local Rankin-Selberg period on the local Whittaker model W e 

w(n„,v^^)(x)W(n„+i,V'£;): 

(3.13) K{W^) = [ WMdh, 
and normalize it by 

^^^^"^ = TTTfr — T' 



19 



The integral is absolutely convergent for tempered representation. Then for unramified data 
we have Xl^{Ww) = vol(i^'^) ([25]). Finally, Jacqquet, Piateski-Shapiro and Shalika showed 
that 

(3.14) A(0) = L(^, n„ X n„+i) Yl \l{w^). 

w 

Decomposing a spherical character. We now denote 

G' = ResE/F (GL„ x GL„+i) 

viewed as an F-algebraic group. We consider its two subgroups: H[ is the diagonal embedding 
of ResE/pGLn (where GL„ is embedded into GL,„+i hj g i-^ diag[g,l]) and H2 is GL„^^ x 
GLn+i^F embedded into G' in the obvious way. 

Now we consider a cuspidal automorphic representation 11 = n„ ® n„+i of G'{A). We 
denote by W(n, ?/;) the Whittaker model >V(n„, V') <S) W(n„+i, iIj). Denote by /? = /3„ (x) (3^+1 
the (product of) Flicker-Rallis period on 11 = n„ n„_|_i. We define a global spherical 
character — a distribution on H{A) — as follows. For /' e ^^"(G'(A)), define 



where the sum runs over an orthogonal basis of 11. Equivalently, 

where the sum runs over an orthogonal normal basis of IT for the Petersson inner product. 
Note that the definition of n(/') involves a choice of the measure on G'{A): we could choose 
any as long as then we use the same measure (quotient by the counting measure on G'{F)) 
to define the Petersson inner product. 

By definition of As-, we have for i = n,n + 1: 

L{s,Ui X nn = L{s,Ui,As+)L{s,Ui,As-). 

Now assume that 11 = vr^; is the base change of a cuspidal automorphic representation vr of 
G{A). By [' , Prop. 7.4] we also have 

L{s,Ili, As^^^^') = L{s,TTi,ad). 

Since 11 is conjugate self-dual: 11 ^ 11'^ where a is the nontrivial element in Gal(E'/F), we 
deduce that 

(3.16) Re...,L(.,n.,AsM)-) ^^^^'"-^^ ^ ) - L{l,.^,ad), 
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where the function L{s,Ili, As^ ^^') is regular and non-zero at s = 1 by a [35, Theorem 5.1] 
(cf. [21, Remark 1.4]). 

We now define a (normalized) local spherical character ioi each place v of F: 



where the sum runs over an orthogonal basis VF„ e >V(nt,, -0^,) and A^, 13^ are the local Rankin- 
Selberg periods via the Whitaker model. 

We now summarize to arrive at an analogue of the decomposition in Conjecture [?] 

Proposition 3.3. Assume that the cuspidal automorphic representation H of G'{A) is the 
base change tie of a cuspidal automorphic representation n of G{A). Then we have 

(3.18) rM') - Lii,„r§^^U'i(a 

Proof. Note that 

Then the result follows from Prop. 3.1, 3.2 and the relation (3.16). 

□ 



4 Relative trace formulae of Jacquet and Rallis 

The construction of Jacquet and Rallis. We recall the Jacquet-Rallis relative trace 
formulae ([2G]) and we refer to [50] for more details. 

First we recall the construction of Jacquet-Rallis' RTF in the unitary case. For / g 
^^{G{A)) we consider a kernel function 

7gG(F) 

and a distribution 

J(/) := Kf{x,y)dxdy. 

JH(F)\H{A) JHiF)\H{A) 

A cuspidal automorphic representation tt of G{A) contributes to the spectral side 

(4.1) Mf) = Y,n<mn^, 

<t> 
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where the sum is now over an orthonormal basis of n with respect to the Petersson inner 
product: 

(^A')= f mW{g)dg. 

Jg(f)\g{a) 

We call the distribution a global spherical character on G (relative to H x H). For a regular 
semisimple S e G{F) and / e C^{G{F)), we define its orbital integral 

0{SJ)= \ f{x-'6y)dxdy. 

JHiF)xH(F) 

We now recall the RTF in the general linear case. Recall that G' = Rcse/f (GL„ x GL„+i) 
as an F-algebraic group. We consider its two subgroups: H[ is the diagonal embedding of 
ResE/FGLn (where GL„ is embedded into GL„+i hj g >-* diag[g,l]) and H2 = GLn^F x 
GLn+i,F embedded into G' in the obvious way. For /' g '^^°°(G"(A)), we define a kernel 
function 

Kf'{x,y)= 2 /'(x^Sy)- 

7SG'(F) 

We then consider a distribution on G'{A): 

I{f) = f f Kf,{h,h2Mh2)dhdh2. 

Jh[{f)\h[{a) Jh^{f)\h',{a) 
A cuspidal automorphic representation 11 of G'{A) contributes to the spectral side 

(4.2) iu{f') = j]^immw^, 

4> 

where = A, ^^2 = Pn® f^n+i correspond the global Rankin-Selberg and Flicker-Rallis 
periods respectively, and the sum is now over an orthonormal basis of IT (with respect to the 
Petersson inner product). This is of course the same as the spherical character we defined 
earlier in (3.17). 

We also recall the definition of orbital integrals for an H[ x iJg-regular semisimple (cf. [49, 
§2]) — for simplicity, regular semisimple: 

(4.3) 0(7,/'):= f f fih^'-fh2Hh2)dhdh2. 

Jh[{f) Jh'^{f) 

Denote by {Hi{F)\G' (F) / H2{F))rsth.e set of regular semisimple {H[ x if2)(-^)-o^^bits in 
G'{F) and {H{F)\G{F)/H{F))rsthe set of regular semisimple {H x i7)(F)-orbits in G{F). 
To show the dependence on the Hermitian spaces W, V, we will write Gwy for G and Hw for 
H. Then there is a natural bijection 

{H,{F)\G'{F)/H2{F))rs - U {HwiF)\Gw,v{F)/Hw{F))rs, 



22 



where on the RHS the disjoint union runs over all pair W <^ V oi Hermitian spaces of 
dimension n and n + 1 modulo the equivalence relation: {W, V) is equivalent to {W, V) if 
there a constant k e such that kW ^ W and kV ^ V (here kW means that we multiply 
the Hermitian form by the constant k). Without loss of generality, we may and will assume 
that V = W®Ee is an orthogonal sum of W and a one-dimensional Hermitian space Ee with 
a norm one vector (e, e) = 1. Moreover, the same holds if we replace F by F^, for every place 
f of F (for the proof, see [49, §2]). 

There is an explicit transfer factor {Qv}v defined in [50, §2.4] with the following property 
on the regular semisimple locus of G'{Fy), indexed by all places v of F, such that 

• If 7 E G'{F) is regular semisimple, then we have a product formula Yiv ^v{l) = 1- 

• For any hi e H[{F^) and 7 e G'{F^), we have Q{hi^h2) = r]{h2)flv{''y). 

We may construct it as follows. Let S'„+i is the subvariety of ResE/pGLn+i defined by the 
equation ss = 1. By Hilbert Satz-90, we have an isomorphism of two affine varieties 

ReSE/pGLn+l/GLn+l^F — Sn+1, 

induced by the morphism between F-varieties: 

u : ResE/pGLn+i -~* Sn+l 
9 ^ 9r\ 

and in the level of F-points: 

GL„+i(F)/GL„+i(F) ^ Sn+i{F). 
Fix an auxiliary character throughout this paper: 

(4.4) 7]' : F"\A^ -> 

(not necessarily quadratic) such that its restriction ?7'|ax = V- Write 7 = (71,72) e G'{F.y) 
and s = i^{'ji^2'^)- If n is odd, we set: 

(4.5) 1].(7) := r/;(det(7i72"'))^:(det(s)-("+i)/2 det(es, es")), 
and if n is even, we set: 

(4.6) fi,(7) := 7;;(det(s)-"/2det(es,...,es")). 

For a place v of F, we say that the function /' e ^^°°(G"(F^)) and the tuple (fw,v){wy), fw,v ^ 
^^{Gw,v{Fv)), indexed by the set of all equivalence classes of Hermitian pair (VT, F) over 
Ey = E ® Fy, are smooth transfer of each other or match if 

nM0{^J') = 0{5Jwy). 
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whenever a regular semisimple 7 e G'{F^) matches 5 e GwyiFy). One of the main local 
results in [50] is the existence of smooth transfer at non-archimedean non-split places (cf. 
[50, Theorem 2.6]) and arbitrary split places (cf. [50, Prop. 2.5]). In this paper, we usually 
need to consider a fixed pair (W, \^) and we say that /' and fw,v ^ '^^{GwyiFv)) niatch 
if there exists some fw',v' ioi all equivalence classes {W, V) 7^ {W, V) such that /' match 
the completed tuple fwy, fwy- Moreover, by [11], the fundamental lemma holds when the 
characteristic of the residue field at v is sufficiently large. 

We need some simplification ([50]). Identify H[\G' with ResE/pGLn+i- Now we write F for 
Fy for a fixed place v. We may integrate /' over H[{F) to get a function on ResE/FGLn+i{F): 

f'{x):=( f'iix,l)h)dh, xe ResE/pGLn+iiF). 

Now assuming that n is odd, so the character rj on H2 is indeed only nontrivial on the 
component GL„+ij;'. Then we may introduce a function on Sn+i{F) as follows: when p[x) = 
s G S'„+i(F), we define 

f\x) := f'{xg)r]'{xg)dg. 

JGL„+i(F) 

Then /' e C^-" {Sn+i{F)) and all functions in C"(S'„+i(F)) arise this way. Now it is easy to see 
that for 7 = (71,72): 

(4.7) 0(7, /') = r7'(det(7i72"')) f ~f\h-hh)^{h)dh, s = 
If n is even, we simply define in the above 

fix) := f f'{xg)dg, 

JGL„+i(F) 

and then 

(4.8) 0(7,/')= f m-'shUh)dh, , = zy(7i7-i). 

JGLniF) 

Up to a sign, the integral on the RHS depends only on the orbit of s under the conjugation 
by GL„(F). Moreover, in either case we have 

^^(7)0(7, /') = mO{s, /'), s = i/(7i72'')- 

A trace formula identity. We are lead to a comparison of the two RTFs and the two 
spherical characters /n and when 11 is the base change of tt. 
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Conjecture 4.1. Let vr be tempered automorphic cuspidal representation on G{A) that admits 
invariant linear functional: 

HomH{A)(7r, C) 7^ 0. 
Let tte be the base change of n. If txe is cuspidal, then we have 

2--'L{l,r])-'l^,{f) = Mf), 

for matching f and f . 

Remark 5. If vr^; is not cuspidal, then n is endoscopic and in the above equahty the constant 
2^ should be replaced by \St,\ appeared in the Conjecture 1.1. All of these should follow from 
the full spectral decomposition of the Jacquet-Rallis relative trace formulae. 

Theorem 4.2. The conjecture 4.I holds if we assume that 

• For at least one split place v, vTj, is supercuspidal. 

• If v\<X), either v is split, or the group G{Fy) is compact and f^ matches the tuple 

0, 0, 0) (with zero functions on every unitary group other than G{Fy)) satisfy- 
ing that: the function f^ is supported in the (open) subset of regular semisimple locus 
matching elements in the compact unitary group G{Fy), and the function f^ is supported 
in the regular semisimple locus of G{Fy). 

Proof. We would like to apply the result from [oO]. But we need to compare the difference on 
the normalization of Petersson inner product in the unitary case (caused by the presence of 
center). There implicitly we use a different Petersson inner product 

<<^, = f mW{g)dg = vol(Z(F)\Z(A))-\0, (/>'>. 

JZ{K)G(F)\G{K) 

Note that the center Z of G is isomorphic to U{1) x [/(I). Hence its Tamagawa number is 

vol(Z(F)\Z(A)) = (2L(l,r;))l 

Now taking into this correction, we apply the trace formula identity [50, Prop. 2.11] to 
matching /' and a tuple {fw) indexed by equivalence classes of {W, V): 

I-KE {n = {2L{l,r^)rJ]J]j^,Mw), 

where the sum is over all equivalence classes of [W, V) and all ttw nearly equivalence to tt. We 
assume that for the {W, V) we start with, fw = / is also the function in the assumption of the 
theorem. Note that by our assumption on L-packet, the multiplicity one holds, namely, each 
TTw are non-isomorphic. Moreover, if an archimedean place v is split, 7rw,v is isomorphic to 
TCy (there is only one member in the L-packet of the general linear group). If an archimedean 
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place V is nonsplit, then the compactness of G{F^) and the assumption on the matching 
and the tuple (/t,, 0, 0, 0) again implies that tiw,v is isomorphic to tt^, (we are using the fact 
that for the compact unitary group, the Vogan L-packet of 7r„ contains only one member). If 
V is a, p-adic place, the result on local Gan-Gross-Prasad in [2] assures that there is at most 
one member in the same Vogan L-packet of vr^ admitting invariant linear functional. By our 
assumption Hom/i-(ir^)(7r„, C) 7^ 0. Hence the sum reduces to one term 

/.J/') = (2L(l,r/))V.(/). 

□ 

Reduction to a local question. Our the main ingredient is an identity between the two 
local distributions. Let dn = n{n — l)(n — 2)/6 be the integer defined by 

r"- = 5n-i{en-i) = det(Arf(e„_i) : Nn-i{E)). 

Conjecture 4.3. Let be an irreducible tempered unitary representation of G'(F^) with 
a„ 7^ 0. Assume that the base change n„ is irreducible generic unitary (so that /n„ is well- 
defined). If the functions f^ s and f'^ e ^^'{G'{F^)) match, then we have 

InXf'y) = K„L{l,r]^y^J^^{f^), 

where the constant k„ is given by 

Here aJn„ is the central character ofUn, and disc{W) g F^/NE^ is the discriminant of the 
Hermitian space W. 

Proposition 4.4. The two conjectures 4-i O'^d 4-3 together imply Conjecture I.4 and 1.1 
when TIE is cuspidal. 

Proof. By Conjecture 4.1 and Prop. 3.3 we have 

.W) - 2-L(l,^)-J.,(r) ^ 2-|[^^n^n„(/:)- 
Conjecture 4.3 is equivalent to the identity between the normalized distributions 

Since 

ne(l/2,??.,^.) = e(l/2,r/) = l. 



26 



we have 

V 

Note that the product Yl^ ^i^iVv)^^^n+i,v converges absolutely to ?7)^-'^A„+i. We thus 
obtain 

Note that the global measure on H and G in Introduction are normalized by L{l,r])^^ and 
L(l,?7)~^ respectively. Correction of measures yields Conjecture 1.4 and therefore Conjecture 
1.1. □ 

We have the following evidence of Conjecture 4.3. 

Theorem 4.5. Let v be a non-archimedean place and tt^, as in Conjecture 4-3. Then the local 
conjecture 4-3 holds if the place v is split in E/F. Assuming that the ratio Iii^{flj)/ JirSfv) is 
a constant, then the local conjecture 4-3 holds under any one of the following conditions: 

• The representation ir^ is unramified, or 

• The place v is non-split and H{F^) is compact, or 

• The place v is non-split and the representation -Ky is supercuspidal. 

Below we first prove the local conjecture when -Ky is unramified or v is split and non- 
archimedean. We postpone the proof of the case v non-split to the last part of §9. When the 
place V is archimedean and split in E/F, we only have an equality in terms of a'y instead of 
a^. That's why in the main Theorem 1.2 we only use a'y. We will prove Theorem 1.2 after 
Lemma 4.7. 



The case of iVy unramified. Then we may assume that 

1. The quadratic extension E/F is unramified at v. 

2. The number r is a t^-adic unit. 

3. The character ifj is unramified and hence so is "0^;. 

Indeed, it is easy to see how /n„ depends on r. And if we twist if) hj a s Fy , it amounts to 
change r by ar. 

We need to utilize the fundamental lemma proved by Yun in the equal characteristic case 
and extended by J. Gordon's to the mixed character case (when the residue characteristic is 
large). We have a matching pair ^ 

^ voi(i7(a))2^^^^''^' ^ voi(//((a))voi(ij^(a))^^'^^^'^' 

^Note that the measures in the fundamental lemma proved in [44] are different from ours. 
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Let be the base change of tv^. Let Wq s W(n„,'0£;) be the unique spherical element 
normalized such that VFo(l) = 1. Then we have 

n (f)w - ^^^(G'm) w 

"^^^ - voi(//((a))voi(//^(a))^°- 

and _ _ 

A(n„(/^)iyo)/3(iyo) voi(g-(a)) \{Wo)/3{Wo) 

""^^^'^ MWo,Wo) vol(//((a))vol(//^(a)) ^?.(H^o,V^^o) ■ 

Note that 

X{Wo) = L{l/2,U,)-Yol{H[{0,)) 

and 

We obtain 

luAK) L{l,7r,,Ad) vol(i7{(a))vol(//^(a)) vol(//„(Os,,))vol(i/„+i(Os,,)) 
In summary we have in the general linear case 

L(i/2,n,) 

""^^"^ L(l,7r,,Ad)- 
In the unitary case, we let 0o ^ ^if" normalized by ||(^o|| = 1- 

voi(G(a)) ^ 
= voi(/f(a))^'^°- 



^.(W^o,W^o) ' ' " ' vol(i7„(Os,,))vol(i/„+i(Os,,))- 

L(i/2, n,) voi(G"(a)) voi(//((a))voi(iy^(a)) 



Therefore we have 



By the unramified computation in [21]: 

L(i/2,n,) 



L{l,n,„Ad)' 
We thus obtain 

vol(G(a)) , L(l/2,n.) 
"""^^•^"^ vol(//(a)) 
Note that A^^-^ is the volume of Kn+i^y for the local Tamagawa measure 

vol(K„+i,,) = L(l,r/)A;i,_,. 

Therefore we obtain that 

, L(i/2,n,) 

We now complete the proof that 
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Change of measures. From now on, all measures will be the unnormalized (namely, with- 
out the convergence factor , L(l, rj^) etc. ) Tamagawa measures with the natural invariant 
differential forms on the general linear groups, their subgroups, and Lie algebras. 

Lemma 4.6. When using the unnormalized measures, the identity in Conjecture in 4-3 be- 
comes 



(4.9) IuM'.) = ^vJ.AQ, 

for matching functions fy and (also under the unnormalized measures). 

Proof. We then see that the old distribution /n„ is the new one times 

where the first term comes from the measure on G' involving the definition of n„(/^), and the 
fraction comes from the measures in A^,/3„ and di,. Similarly, the old distribution J^^^ is the 
new one times 

where the first term comes from the measure on G involving the definition of TTi,{fy), and the 
second from the measure on H{Fy) in the definition of Moreover, the change of measures 
on H[{Fy) , H2{Fy) and H{Fy) also change the requirement of smooth matching: if and 
match for the normalized measures, then C,E,viX)CF,v{X)'^ fv and L{l,r]yYf!^ match for the 
unnormalized measures. Therefore, when using the unnormalized measures, the identity in 
Conjecture in 4.3 becomes the asserted one. □ 



The case of a split place v. Assume that F = Fy is split. Let tt = 7r„ n^+i be an 
irreducible unitary generic representations of G{F). We may identify Hn{E) with GL„(F) x 
GL„(F) and identify U{W){Fy) with a subgroup consisting of elements of the form {g,*^ g~^), 
g e GL„(F) and *(? is the transpose of g. Let pi,p2 be the two isomorphisms between U{W){F) 
with GL„(F) induced by the two projections from GL„(F) x GL„(F) to GL„(F). If vr^ is 
an irreducible generic representation of U{W){Fy), the representation n„ = BC{TXn) can be 
identified with pXti^ ®P2'^„ of Hn{E) where p*vr^ is a representation of GL„(F) obtained by 
the isomorphism pi. For simplicity, we will write n„ = 7r„ (x) Similarly for tt^+i, n„+i. We 
fix a Whittaker model >V(7rj), i = n, n + 1 using the additive character if) at v. Recall that in 
the Introduction we have defined an alternative element a' e Hom(7r (x) yr, C) by 

a'{W, W) = X{W)X{W'), W, W G W{n). 

Again we have identified vf with vf. Moreover, we may require that the invariant inner product 
on TT is the one defined by (cf. §3). 
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Lemma 4.7. Let f = h®f2^ K"{G'{F)) and f = f, . s ^^{G{F)) (f*{x) = f,{x-') 
matching /'. Then we have 

Moreover, Theorem 4-5 holds for split v. 

Proof. We identify n„ with 7r„ (g) 7r„ = 7r„ (g) 7f„. Then we have for W, W g >V(7r„): 
(3niW(g)W) = f w( ^"-''^ ^""'^ ] dh. 

JNr,-l{F)\Hn-l{n V / V / 

This yields 

Similarly for Then the desired equality follows by the definition of In in terms of the 

linear functional A,/3 and i!) (note that 5 = is indeed a square in F). Now Theorem 4.5 
follows from uju„{t) = ■ <^nn{'^) = 1 since the central character u^^ = u~\ □ 

we now give the proof of the first part of Theorem 1.2. 

Proof of Theorem 1.2: the case (1). If for some place v the linear form a„ vanishes, then both 
sides vanish and Theorem 1.2 is trivial. Then by the multiplicity one result in [2], there does 
not exist non-zero iif(A)-invariant linear functional on any other member different from tt in 
the Vogan L-packet of vr. It follows from [50, Theorem 1.1] that if the linear functional ^ 
vanishes on vr, the value L{1/2,'Ke) must vanish, or equivalently ^(1/2, vr^;) = (since the 
adjoint L- function does not vanish at s = 1). Hence we may further assume that the linear 
functional does not vanish on tt and =Sf (1/2, tt) 0. Under the condition of Theorem 1.2, 
Conjecture 4.1 holds by Theorem 4.2: 

2-'L{l,ri)-'h,{f') = Uf). 

It follows that the both sides are non-zero distribution. A prior we know that 

uf) = ^.WJiSf^). 

V 

Therefore for every v, In^{f[,) differs from Jn^{fv) by a non-zero constant independent of the 
matching functions fy and It follows from Theorem 4.5 that for all non-archimedean v, 
the identity in Conjecture 4.3 holds for i;|oo when we replace J-j^^ (using ay) by J^^ (using a'^). 
Since all v\co are split in E/F, by Lemma 4.7, the identity also holds for w|oo. Now Theorem 
1.2 follows from the proof of Prop. 4.4. □ 

Now we assume that F is non-archimedean. We would like to identify the distribution 
(defined using a') with (defined using a). Note that in the definition of a, the inner 
product is now defined by ■!?: 

a{W, W) = ( (7v{h)W, W'}dh, (W, W) = d{W, W). 
Jh{f) 
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And the measure on H{F) ^ GL„(F) is the unnormahzed Tamagawa measure. Let c 6 M_|_ 
and let A^'^ be the subgroup of Nn{F) consisting of u such that < c, z = 1, 2, n — 1. 

Lemma 4.8. Assume that iTn is tempered. For W, W s W(7rn,), then the integral 

f (nMW,W'}i;{u-')d 
Jn'=(f) 

stabilizes as c ^ co. We define 

rreg 

^(^W,W'):=\ (7rr,{u)W,W')ip{u-^)du 
Jn„{f) 

being the limit as c —* co. Then we have 

^{W,W') = W{1)W{1). 

Proof. The LHS does not vanish identically by Waldspurger [iO, §3.5]. Both sides define 
non-zero elements in 

Hom7v„(F)(W^„, ^) (X) Hom^„(F)(>V^> 

By uniqueness of Whittaker functional (up to scalar), the lemma holds up to a unique non-zero 
constant multiple. Then the ratio is evaluated by Lapid-Mao. □ 

Proposition 4.9. Assume that tt = 7r„ (g) iVn+i is tempered. Then we have, for all W, W e 
>V(7r); 

a{w,w') = \{yv)\{y/'). 

Namely, a = a' as non-zero elements in Hom(7r (x) tt, C). 

Remark 6. This should also be true for F archimedean. But some convergence issue has not 
been solved. In practical application, it seems that the linear form a' is more suitable for 
computation that a. 

Proof. The RHS does not vanish by the non- vanishing of local Rankin-Selberg integral ([25]). 
By the multiplicity one theorem, the LHS is a constant multiple of RHS for all W, W. Hence it 
suffices to prove the identity for some choice of W, W. Let W = W„(x)iy„+i, W = VF^(x)W^+p 
We choose Wn+i,W^^i such that their restriction to Hn{F) has support compact modulo 
Nn{F). More specifically, we fix an open (relative) compact subset S c A„(F)A„_(F) we 
may choose Wn+l,W^_^^l such that their restriction to Hn{F) has support inside Nn{F)E. 
Then we have, for g e Hn{F), 



Wn+i{xg)W^_^^{x)dx 
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for a suitable measure on S. Let c e M+ and consider the subgroup A^*^ of A^„(F). Let us 
consider the integral 

UW,W'):= f f f <i>w„,wd^-'ug)i,{u)Wn+i{g)W[^,{x)dudxdg. 

JS JE JNc 

This is equal to 

By [40, §3] for non-archimedean case, the integral 

(4-10) ^^^^g)w„,n„{x)wd^)i^i'^)(^'^ 

converges absolutely. Its proof also shows that the integral defines a continuous function of 
both X and g. In particular, the integral 1^ converges absolutely. Moreover, the limit of 
IciW-i W) as c goes to oo exists and is equal to 



T^n{g)Wn{l)'Kn{x)W^{l)W^+MW^^^{x)dxdg. 

Indeed, when F is non-archimedean, for each {x,g), the integral in 4.10 stabilizes when c is 
large enough. But since x, g vary in a compact set S, this constant c can be chosen uniformly. 
Therefore the limit limc_»oo Ic{W, W) is equal to 

a^g)Wn, 7T{x)W:,)Wn+i{g)W[;;{x)dxdg. 

By Lemma 4.8 we have 

Wr,{g)Wl{x)Wn+,{g)WUii^)dxdg. 

This is precisely X{W)X{W'). 

There is another way to evaluate the limit. Since the integral is absolutely converge, by 
Fubini theorem we could rewrite the integral Ic as 

Ic{W,W')= f f ^w„,wd^-'g)Wn+i{g)W[:^,{x)dgdx. 

Now we could make a substitution g t-^ xg: 

Ic{W,W')= f f <^w^,wd9)Wn+iMW[;;{x)dgdx. 
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Again by Fubini theorem we have 



JEN'^E 

Now by the absolute convergence of 

^Wn,wd9)^w„+i,w'{9)dg, 

and supp{(^\Yn+i,w'„^^ ^ '^Nn{F)E, we conclude that limc_^oo Ic{W, W) is equal to a{W, W). 
Finally, we need to verify that under the restriction of support of W, W, we may still find 
some choice such that X{W)X{W') does not vanish. But this is clear since we are free to 
choose S, Wn and arbitrarily. □ 



5 The totally definite case 

We now prove the part (2) of Theorem 1.2. We hence assume that G{Fco) is compact. 
Equivalently, F is a totally real field, E' is a CM extension and the Hermitian spaces W, V 
are positive definite at every archimedean place v of F. As in the proof of the part (1) of 
Theorem 1.2, we may assume that the linear functional <^ does not vanish on vr and 7^ 
for all V. 

Since G{Fy) is compact for every v\gc, we may choose a function supported in the 
(relative to Hi,) regular semisimple locus. Then there exists a function e G'{Fy) which 
matches (by our convention, this means that for all other isomorphism classes (V14,K)), 
the functions are chosen to be zero). For non-archimedean places we have no restriction of 
choice of test functions. For such matching functions /' and /, by Theorem 4.2 we have 

2-2L(l,r7)-2/,,(/') = J.(/). 

For ti|oo, as the representation tt^, is finite dimensional and Homjy^(7r^,, C) 7^ , clearly we 
may choose an supported in the regular semisimple locus so that J-nSfv) ^ 0. Now we may 
choose /°° = ®v«x,fv so that J7r(/) does not vanish. Then for each f |oo, there is a constant 

7^ depending only on 7r„ such that 

Now we don't know how to evaluate 6^^^. Nevertheless the same argument as the proof of 
Prop. 4.4 shows that 
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where the constant c.„^ = Ylv\aD "^t^ ^^"^ 
where i^D IS the constant in Conjecture 4.3. 

6 Harmonic analysis on Lie algebra 

We estabhsh some basic results to prove the identity between local characters, Theorem 4.5 
for a non-split non-archimedean place v. 

Relative regular nilpotent elements on M„+i(F). The group Hn-, viewed as a subgroup 
of Hn+i, acts on Mn+i by conjugation: 

:= hXh-\ X e Mn+uhs Hn. 

Write 

The ring of invariants is generated by 

trA*X, e*+iX^e„+i, 1 ^ i ^ n + 1,1 ^ j ^ n; 

or 

tr A* A, vA^u, w, 1 ^ i ^ n,0 ^ j ^ n — 1. 

The i/„-nilpotent cone A/" is defined to be the zeros of all of the above invariant functions 
on Mn+i- An element in Mn+i is called Hn-regular (or regular if no confusion arises) if its 
stabilizer is trivial. Denote 



/O 1 





o\ 





1 





... 





1 


\q ... 





0^ 



e M„+i(F), 



and its transpose. Clearly ^+ are regular nilpotent. 
Lemma 6.1. Let ^ g M . The following are equivalent. 

1. ^ is regular nilpotent. 

2. ^ is Hn-equivalent to 

3. A+(O^0. 
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Similarly we may replace + by — 



Proof, li X = \^ y //n-nilpotent, we have = and vA^u = 0, for i = 0,1, n — 1. 

Let r be the dimension of subspace of the column vectors spanned by A^u, i = 0,1, ...,n — 1, 
and similarly r' the dimension of the subspace spanned by vA^, i = 0,1, ...,n ~ 1. Clearly, we 
have r + r' = n. we need to show that if X is regular unipotent, then either r or r' is equal to 
n. Indeed, for example if r = n, then the column vectors A^u, i = 0,1, ...,n — 1 form a basis 
of the n-dimension space. In term of this new basis we see that X becomes 

Suppose r, r' < n. Clearly ii r = r' = 0, X must have positive dimensional stabilizer 
hence non-regular. Without loss of generality, we may assume that < r < n. Let L be the 
subspace spanned hj A'u,i = 0, 1, r — 1 (it is easy to see that this is the same as the space 
spanned by A^u, i = 0, 1, n — 1). And we write the column vector spaces F"- = L® L' for 
a subspace L'. Then in term of the basis of L given by A^u, i = 0,1, ...,r — 1, we may write u 
as (0, 0, 1, 0, 0, 0)* where only the r-th entry is one, and 

Then A^u = (0, 0, 1, 0, 0, 0)* where only the (r — i)-th entry is one, i = 0,1, ...,r — 1. 
Hence the conditions vA^u = implies that v is of the form (0, 0, 0, *, *) where the first 
r-entries are all zero. 
Now we consider 

li,)^GL„(n 

Clearly the action of h takes X to an element of the same form with B replaced by B+XY—ZX. 
Hence the stabilizer of X at least contains all h with A satisfying Ay — ZX = 0. Clearly there 
exists a positive dimensional of such A since < r < n. We are done. 

□ 



/ 1 

1 

... 

\ ... 



\ 



1 

/ 



G Mr{F) 



Lemma 6.2. Let F be any field. Denote by V the subvariety of Mn^F consisting of X that is 
nilpotent (i.e., X"' = 0) of the following form: 



X 



* 1 

* * 1 

* 1 

* ... * * 



E M 



Denote by V the subvariety of Mn,F consisting of X the same form but with the last row 
identically zero (not assuming unipotency of X). Then we have a natural projection p -.V ^ 
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v. Define an F -morphism 



u 



V 



and denote p' = po v : V . Then p, u, u' are all isomorphisms between affine varieties. 

Moreover, the Jacobian of u' is +1. 

Here the Jacobian is in term of the obvious coordinates on and V. Similarly in the 
proof below. It follows from the lemma that X is conjugate (under a unique element in 
N^{F)) to In particular X is regular unipotent (the regularity also follows from the fact 
that A^e spans the entire space of column vectors). 

Proof. It is easy to see that u^+u~^ lies in V (this also follows from the proof below ). We 
prove this by induction on n. It is trivial when n = 1. We use Nn_,Vn etc. to indicate 
the dependence on n. We naturally consider A^^.i „ as a sub variety of A^n,-- We may write 
Un e as a product u ■ Un-i where Un-i e Nn-i- and u is of the following form 



(6.1) 



u 



Xi 









, Xi G F. 



Clearly Un-i = u ■ Un-i defines an isomorphism of affine varieties Nn- = A^n-i, 
Jacobian +1 (in terms of the obvious coordinates). 

We write = Ur,.-^^r,-^.+u:^\^ e Vn-i, x = {xi, ...,Xn-i) e F""^. Then 



X F"-^ with 



-lCn-l, + Un-i £ Vn-1, X 
t -1 



U 







u 



is given by 



xXn^l 



.iX 



Replacing x by —x + where x„_i is the last row of Xn-i, by induction hypothesis that 
is an isomorphism, we conclude that so is u'^. The formula also shows the Jacobian of 
u'^ is +1 by induction hypothesis. 

It remains to show that pn is an isomorphism. We will define an inverse of p„ inductively. 
We start with X of the form 



X 



n-l 



ai 





1 

-1 fln 



n-l- 
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We consider a unipotent u in N_{F) of the form as in (6.1). Then we see that the most lower 
right entry of uXu~^ is given by a„ + Thus we may choose Xn-i = —ctn and hence 

assume that in X we have a„ = and in u we have Xn-i = 0. Then inductively if we have 
ttn = ttn-i = ... = ttn-i = 0, and Xn^i = Xn~2 = ... = Xn-i-1 = 0, then (using the property of 
Xn-i) we see that the {n,n — i — l)-th entry of uXu^^ is given by a„_j_i + Xn-i-2- Therefore 
we have (a unique such) u of the above form such that uXu~^ is of the form 



uXu 










1 





where retains the same shape as (by computation). Note that 

det(X) = +(ai - an * det(X„_i)). 

Now it is easy to see that = det(X) = det{uXu'~^) = +{a'i — 0). Hence a'l = 0. Moreover 
the unipotence of gXg^^ (being block upper triangular) now implies the unipotence of 
Now it is clearly that by induction hypothesis the above process defines an inverse of Pn. 

□ 

We also have an similar but easier statement on the upper unipotent A^+. 

Lemma 6.3. Let F be any field. Denote by V+ the subvariety of Mn^F consisting of X that 
is nilpotent (i.e., X" = 0) of the following form: 



X 



Define an F -morphism 



1 


... 



v+ : iV+ 
u 



1 * 
1 




n,F 



Then v+ is an isomorphisms between affine varieties. Moreover, the Jacobian of v+ is +1. 
Proof. Similar to the previous one. We omit the detail. □ 



Regular nilpotent orbital integral. Let F be a local field. We now define the {Hnyt])- 
orbital integral of a regular nilpotent orbit. For X = \ ^ ^ ] e M„+i(F), we define a 



V w 



matrix 



5+{X) := (A"-^m,A"-'m,...,m) e M„(F) 
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and its determinant 
Similarly, we define 
and 



A+(X) = det(5+(X)). 
S^X) := 6+{X'), A_(X) = det(5_(X)), 



A := A+A_ 



Since the orbits of ^+ are not closed, we need to regularize the orbital integral. We consider 
for s E C: 

0{X,f,s) = v{^4X)) f f{X'^Mh)\det{h)\'dh. 

Jh„{f) 

Lemma 6.4. The integral 0{^+, /, s) converges absolutely when Re(s) > 1 — - and extends to 
a meromorphic function on s e C with at most a simple pole at s = 1 — 1/i, for even integers 
i with 1 < i ^ n. 

Proof. We use the Iwasawa decomposition of Hn{F) = KAN. Define 

fxiX) = f f{kXk-^)dk. 

JK 

By Iwasawa decomposition on Hn{F), we have 

fK{au^+u^'^a^^)r]{a)\a\'^\5{a)\duda, 

IA{F) JN{F) 

where the modular character 

d{a) = ai^^a2^^ ■.■an^^''^^\ a = diag[ai, ...,an]- 

By Lemma 6.3, this is 

fKiaxa^^)ri{a)\a\'^dxda 

■JA{F) Jv+{F) 

Replacing Xij by XijOja^^, 1 ^ i < j — 2 < (n + 1) — 2 and setting a.„+i = 1, we may cancel 
partially the factor |5(a)|: 



Ja{f) Jv+{f) 



^ Xi3 Xi4 * 

0^** 

as 



a„ 

... 



ri{a)\a\'^\a2a3...an\ ^dxda 
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Substitute bi := aj/aj+i, i = 1,2, n ~ 1, &„ = a„: 



Ja(f) Jv+(f) 



bi xi3 xm * 

^2 * * 



6„ 

... 



v{b,b,...)\Ylbl\-^+'Yl'^kdx. 



1=1 



Now it is clear that the integral converges absolutely if Re(2(— 1 + s)) > —1 for all i = 
l,2,...,n, or equivalently Re(s) > 1 — ^. And by Tate's local zeta integral, the integral 
extends meromorphically to s e C with at most simple poles at those s such that 

i{-l + s) = -l, i = 2,A,...,2[n/2]. 

Namely s = 1 — 1/z, for even i with 1 < i ^ n. 

□ 

Definition 6.5. For f e "T^^lMn+i), we define the regular nilpotent orbital integral 0{^+, f), 
also denoted by ^£_+{f) as 

/^€±(/) = O(e±,/):=O(e±,/,0). 
This defines an {Hn, r])-invariant distribution on Mn+i- 



A regular section of or. Denote 

M„+i,+ :={XeM„+i|A+(X)^0.} 

Then X e Mn+i,+ if and only if 5+{X) e GLn,Fi or equivalently u,Au, ...,A^~^u are linearly 
independent. The latter property implies that the stabilizer (under the i/^-action) of any 
X e Mn+i,+ must be trivial. Indeed, if hXh~^ = X, we have hAh~^ = A,hu = u. Therefore 
hA^u = hA^h~^ ■ hu = A^u for all z = 0, 1, ...,n — 1. It follows that h = In from the linear 
independence of u, Au, A^~^u. 

We will write ^ = A" x A"+^, ^ the affine space of dimension 2n + 1. Then the second 
set of generators defines a morphism that is constant on J/„-orbits 

TT : M„+i ^ = A" X A"+\ 
\ V w J 

^we only use here A to denote the afBne line. 
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where a = (ai, a„), b = {bo, ...,bn), flj+i = tr a*+^ A, bo = w and &j = vA^u for < z n — 1. 
Now we define a section of the morphism n : M„+i ^ 













/ CLl 


1 








\ 







1 
















1 
















1 








bi 


bo ) 



We note that ^+ is precisely the image of e ^ under a. 
Lemma 6.6. The morphism a is a section of n, i.e.: 

a o n = id. 

The image of a lies in M„+i + (in particular, a is a regular section, in the sense that the 
image a{a, b) is always Hn-regular). 

Proof. Indeed it is easy to check that 



det 



T + 



V 



/ ai 1 \\ 

as 1 

... 1 

\an y / 



T" + aiT"-i - aaT"-" + ... + (-1 



n-2 



\n-l. 



and the fe-invariants of a{a,b) are (6o, &2, ^n)- This shows that a is a section of n. To 
see that the image of a lies in M„+i^+, we note that for any (a, b) e ^ we have 



□ 



6+{a{a,b)) = 1„. 

Note that 6+ (restricted to Mn+i,+) defines a morphism 

S+ : Mn + 1,+ ^ GL„ 

X^d4X). 

Moreover, it has the property 

6+{hXh-^) = hS+{X), h e GL„,j.. 

Proposition 6.7. We define an Hn-equivariant morphism 

L : GL„ X -> M„+i,+ 
(/i, (a, b)) ^ ha-{a, b)h^^ , 

where Hn-acts on LHS by left translation on the first factor, trivially on the sec ond. Then l 
is an isomorphism with its inverse given by {5+,tt\^^). 
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Proof. It suffices to prove that n) o l = id and l o tt) = id. To show the first identity 
we note that the invariants of ha{a, h)h~^ (being the same as a{a, h)) are (a, h). Hence it is 
enough to show that S+{L{h, (a, b)) = h. This follows from the fact that 5+(cr(a, b)) = 1„ and 

I A u 

6+{hXh^^) = h6+{X). Now we show the second identity. Let X = 



V w 

n-2 



u,...,u). Denote 



Let (a, fe) = 7r(X) be the invariants of X and h = S+{X) = {A"'^^u,A 
(A v! \ 

io 7r)(X) =1 , , I . Clearly w = w' . By the first identity, the elements 7r)(X) 



V w 

and X have the same invariants. In particular, 



det(T + A) = T" + 2( 



and 



This implies that 



A"" = Y,a^A''-\ 



i=l 



A5+{X) = A 



.,Au) = S+iX) 



(ax 1 \ 

as 1 

... 1 

V an / 



Since 5+(X) = /i is invertible, we obtain 



A = h 



Obviously we have u = S+{X)en = hCn 
{bn,bn-i, ...,bi) = v6+{X), we have 



/ 


ai 


1 





^ 




a2 





1 















1 


V 


a-n 











u 



h-^ = A'. 



[en = (0, 0, 0, 1)*). Finally since k = vA^'^u, 



V = {bn,bn^i, ...,bi)S+{X) = (bn,bn-i, ...,bi)h ^ = f '. 
This completes the proof of the second identity. 



□ 



Now we apply the results above to the orbital integrals. Assume that F is a non- 
archimedean local field. 

Corollary 6.8. The intersection M„+i +(F) nA/" is equal to the Hn-orbit of In particular, 
for a function f e ^^°°(M„+i(F)) supported on Mn+i,+{F) , any distribution on M„+i(F) 
supported in the closed subset N\(Hn ■ ^+) of Mn+i{F) vanishes on f. 
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Proof. Since f+ is precisely the image of e ^ under a, the i/„-orbit of ^+ is then the image 
of Hn X {0} under l. We also have M„+i + n A/" = (7i"U/„+i.+ )~^(0), the fiber of e ^ under 
7r|M„+i.+ - By Proposition above the fiber (7r|jv/„+i_+)^H0) is precisely the image of Hn x {0} 
under i. This proves the first assertion. The "In particular" part is clearly from the definition 
of support of a distribution. □ 

Proposition 6.9. For any f g ^^'°(M„+i,+ (F)) c <^*(M„+i(F)), the orbital integral 

Mx) ■■= Oiaix)J) 

defined for regular semisimple x s ^ extends to a locally constant function with compact 
support on ^ (i.e., (pf e '^^^(^). And conversely, given any function 4> in '^^{^), there 
exists f e ^^°°(M„+i^+(F)) such that 0((t(x), /) = for all regular semisimple x. 

Proof. Since A+(o"(a;) = 1, the normalized orbital integral is given by 

0{a{x)J)= [ f{ha{x)h-^)ri{h)dh. 

By the i/„-equivariant isomorphism i : Hn{F) x ^{F) ~* Mn+i,+{F), corresponding to / we 
have an element denoted by /' in ^^{Hn x ^) with the property that 

0{a{x),f) = [ f'{h,x)rj{h)dh. 

The integral in RHS is clearly convergent absolutely for all x e and defines an element in 
^^{^). The converse is clearly now by the isomorphism t. 

□ 

Remark 7. The proof also shows that if / is supported in M„^_i(F), the integral 0{^+,f,s) 
converges absolutely for any s e C. 



Harmonic analysis on s„+i. We also need to consider the induced if^-action on the 
tangent space s„+i at 1 of the symmetric space Sn+i- 

5„+i(F) = {X E M„+i(F)|X + X = 0}. 

For any choice of non-zero r e E~ , we have an isomorphism 

Mn+l{F) ^Sn+l{F), 

defined hj X i-^ tX. In particular, we will abuse the notation ^+ to denote if we want 
to consider the regular unipotent orbit on Sn+i- Then clearly every result established in this 
section holds for Sn+i- 
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7 Smoothing local periods 



Convolutions. We introduce some abstract notions for the use of this and the next section. 
Let F be a p-adic local field and G the F-points of some reductive group. We consider the 
space of 'lo^iG) with an (anti-) involution * defined by: 

We will also use the other (anti-) involution defined by 

rig) = fig-')- 

Let dg be a Haar measure on G. Let H he a closed subgroup of H, unimodular with a Haar 
measure dh. We define a left and a right action of ^^\H) on '^^{G) as follows: for / e ^^{G) 
and (f) e '^^(H), we define convolutions / o and o / both in '^^{G): 

ifo<jy)ig)= f figh-')mdh 
Jh 

and 

i<po f){g)= r <P{h)f{h-'g)dh= ( <P{h-')f{hg)dh. 
JH JH 

This also applies to the case H = G. Then we have 

(/o0)* = 0*o/*, (0o/)* = ro0*. 

If we have two closed unimodular subgroups Hi,H2, we could iterate the definition: for 
example, for / e and (pi e "^^{Hi), we define 

0io02o/:=0io(02o/)e^^=^(G'). 

Now if we have a smooth representation tt of G (hence its restriction to if is a smooth 
representation as well), as usual we define vr(/) and vr(0) to be the endomorphisms of tt: 

^(/) = f fig)^g)dg, 7r{<P) = f <pihMh)dh. 

JG JH 

Then we have 7r(/o0) = 7r(/)7r(0) g End(7r) and so on. If tt has an G-invaraint inner product 
(■, •), we then have 

<7r(/)M, m'> = <u, 7r(/*)u'>, u.u's-K. 

The question addressed in this section has the following shape. Let tt be a smooth admissible 
representation of G with an invariant inner product. Very often the algebraic dual space 
TT* := Hom(7r, C) is much larger than the congragredient vf (the subspace of tt* consisting of 
smooth linear functional, i.e., finite vectors in tt* for some open compact K). Very often 
we will be interested in some distinguished element called ^ in 7r*\7r (the set of non-smooth 
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linear functionals) . It is also very often useful to find some e 'io^{H), for suitable subgroup 
H (smaller than G in order to be useful), such that 7r*(0*)£ is indeed smooth (and non-zero), 
i.e., there exists 7^ e tt such that for all -u e vr: 

£(7r((/))M) = {u,u'). 

We will encounter this kind of questions for the local Flicker-Rallis period, and the local 
Rankin-Selberg period. 

A compactness lemma. Now we return to our setting. Let E/F be a quadratic extension 
of non-archimedean local fields of characteristic zero. Let 77 be the quadratic character asso- 
ciated to E/F, and denote r]^ = v""'^- Let ipn-i e "tf^ {Hn-i{E)) and (j)n-i e ^,°°(M„_i,i(^)). 
We consider the Fourier transform of (pn-i as a function on the Mi^n-i{E) by 

= \ 4>n-i{Y)^l^E{ti{XY))dY. 
To the pair (v?n-i, 0n.-i) we associate a new function on Hn-i{E) by 
(7-1) Wvr.-i,K-ii9) ■= (Pn-ii~el_^g) J(/J„_i(^"^we„_i/i)^/'^(u)r/„(/i)duc//i, 

where u e Nn^i{E),h e Nn-i{F)\Hn_i{F) . Clearly we have 

for u E A^„_i(^). 

We would like to obtain a function with support compact modulo Nn-i{E) by imposing 
some conditions on ipn-i, 4>n-i- We construct a subspace of special functions by the following 
procedure. Fix integers /c ^ 0, m > 0. Define a sublattice of Oe 

(7.2) A = Afc := zo'Oe. 

It decomposes as = A^ © A^ where A^ = tz7^(9£;+. Then the Fourier transform of 1a e 
^^°°(i?) is a non-zero multiple of 1a* for a lattice A* c i? (depending on i/j). We naturally 
view C['cu"^A/zu'^"^A] as a subspace of ^/^{E) consisting of functions supported in w^A and 
invariant by w^'^A. We define a "dagger" space of level (/c, m), denote by C[w"*Afc/ti7^'"Afc]^ or 
K°^(^)I,m ' as a subspace of C[w'^k/w'^'^k] spanned by functions 6 = 9+^0-, 6^ e ^c°°(£;±) 
satisfying: 

• 6'+ is a multiple of lromA+- 

• The Fourier transform e C[w~^'^A*/zi7~™A*] is (non-emptyly) supported in w^^'^A* — 
^-2m+i^*_ Equivalently, Q- is supported in ^"^'"A"* - ti7"^'"+^A"* where A"* = 
A* n 
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In particular, every element in C[uj"^A/w'^"^A]^ is invariant under multiplication by I + w^^Oe- 
We also denote by ^^°°(M„_i i (£'))[, ^ the space of functions on M„_i_i(E') of the form 0„_i = 
(8)issi=£n^* ill the way that </)„_i(a:) = Yli^t^^^i) if a; = (a;i, e M„_i_i(i?) satisfying 

• When 1 < 2 < n — 1, 0* is the characteristic function of g7"*A; 0" is an element of 
C[w™A/ro2™A]t. 

We denote by ^^{Hn-i{E))\^ the space of functions on Hn-i{E) of the form ^pn-i = 
®i^i,j^n-i ill the way that = Oij V5*^(fi'ij) if = (^ij) satisfying: 

• When 1 < z, J < 77, — 1, j — z 7^ 0, 1, v?*-' is the characteristic function of w^A. 

• When l<i = j^n — 1, (p^^ is the characteristic function of 1 + w^K. 

• When 1< 2 = J - 1 s$ n - 2, y?*Ms an element of C[u7'"A/u72'"A]t. 

Let ^,°°(M„_i,i(£;))|„ be the union of ^,"(M„_i,i(^))^ for all k ^ 0. Similarly for ip^_i e 

'^^{Hn-i{E))l^. We say that the pair ((/^n-i, 0n-i) is m- admissible ii <pn-i ^ ^c°°(^n-i,i(-£^))L5 
and tpn-i £ {Hn-i{E))l^. We also call them "dagger" space of level m. 

The pair (v9„-i, 0„-i) defines a function denoted by (pn-i ® 0n-i on the mirabolic group 
Pn of Hn{E) with support in 1 + p„(ro'"Afc) where p„ is the Lie algebra of P„: 

= ¥'n-l(2;)0n-l(^i)- 

Some property of admissible functions will not used until next section. Our key lemma of 
this section is as follows. 

Lemma 7.1. Assume that {^Pn-ii4>n-i) is m-admissible for some m > 0. 

(1) Then the support of the function W^<^^_i,(/)„_i is compact modulo Nn-i{E), i.e., it defines 
an element in 

^^{Nn-i{E)\Hn-i{E),i^j,). 
More precisely, W^p^_^^^^^_-^{en-lg) is nonzero only when 

g E K_,{E) = Nn-i{E)An-i{E)N^.i,.{E). 

(2) For i = n ~ 2, we define recursively (pi, (p^^^ such that 

where 

Set = ipi so that we have the following pattern of decomposition of (pn-i ® (pn-i- 



V^n-l (H) (pn-i 



X 



Ln-l 



U 
1 
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01 01 


02 




0n-l 


02 




0Ll 



Set (j)' = ®"riVi viewed as a function on Bn-i-{E) or hn-i,i{E). Denote by dn = 
n{n - l){n - 2)/6. Then the value o/ W<^„_i,0„_i(e„_i^) when g = yz s A„_iiV„_i _(F) 
for 

( V\V2---Vn-\ \ 



and 



n-2 



is given by the product of a constant 

\ (f)'{b)db 
where r'^" = 5„_i(e„_i) = det(Ac/(e„_i) : Nn-i{E)), and 



n-l 



(7.3) 



'n{y)\5n-l{y)\F W (j)n-i{~yi{Zn-i-l, 1)t) . 



j=l 



Here the measure db on is either the left or the right invariant one; they give 

the same value to the integral since the support of 0' is in 1 + whn-i-{OE)- 

Proof. It suffices to consider the following function: 

w{g) = ip{g'^uh)'i{j^{u)r]n{h)dudh, 

where 

i/jriu) = iPe,t{u) = ipE{en-iue:^^i). 
Indeed, by a suitable substitution we have 

W^,^^_,{en-ig) = \T\'j^}n~i{-e^_^Tg)w{g). 
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By the condition on (pn-i, (f'n-ii^n-ig) is zero unless the (n — l,n — l)-th entry of g is 
nonzero. Up to the A'„_i(£')-action, such g e Hn^i{E) is of the form: 



By the support condition on (j)n-ii for w{g) to be nonzero, \yi\ hes in a compact region of E 

ln-2 

Hence we have 



and ||-Zn-2|| is small enough so that Lp is invariant under left multiplication by 



(7.4) M^^„_,,0„_i(en-i^7) = |r|^"0„_i(r2/i(2;„_2, l))w 



yi I I 



Therefore it is enough to consider w{g) when g = Vi ^ for x„,__2 ^ -f^n-2 (-£')• 

The integration on h e Hn_i{F) can be replaced by the integration over then open dense 
subset K_,{F) = iV„_i(F)A„_i(F)iV„_i,_(F). We write h e H'^_,iF) as 

^ _ ln-2 <_2 ^ f an-2 \ f 1^-2 



1 7 V 1 y V Cn-2 1 

where h e F"", a„_2 e H'^^^i^), c„_2 e Mi,„_2(-^), <_2 ^ M„_2,i(-^)- The measure on H'^_^{F) 
can be decomposed as 

dh = \an^2\'^\bi\^^dhidcn-2dan-2dd^_2- 



We write 

u 



In the decomposition of we may ignore the factor in Nn-i{F). Then the product g ^uh is 
equal to 

, -1 / ^rt-2 ^ \ ( ln-2 'L'n-2 ^ / ^n-2 ^ ( <^n-2 \ ( ln-2 



1/ V i/V lyV i/V 1 7 ■ 

Then the last row of the product is equal to yi^bi{cn-2i 1) ^ Mi^n-i{E). By the condition 
on (fi'n,_i, we can assume the ||c,i_2|| is small enough so that tpn-i is invariant under right 

translation by f "'""^^ , | . The product of the first four matrices is then 

V ^^-2 1 y 

-It f Xn-2'^n-20'n-2 Xn-2'^n-2 

The integrations on c„_2 and Vn^2 yield respectively 



0n-l(?/l ^f'l(Cn-2, 1))C^C„_2, 

JA/„_2,i(i^) 
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-2 



(here we note that the Fourier transform of 4>n-2 is defined by the character iI)e-) Therefore 
w{g) is equal to the integration of the function of 61 e given by the product of the above 
two terms and 

fn-2{yi h\Xj^_2Un—2Q-n—2 )V'r(Wn-2)|an-2| ^ T] (h) dUn^2dan 

with respect to the measure \bi\^"'^^^^d*bi. We may repeat the process and hence may assume 
that the function on Hn-2{E) mapping gn-2 to 

0n-2(-e„-2fi'n-2) V5n-2 (^„"-l2^ri-2an-2) V^r (^n-2) I ^71-2 T^??™ (a) C^^^n-2'^a 

JAf„_2(F)W„_2(F) JNr,-2{E) 

is nonzero unless g2 e H'^_2{E) and its support is compact modulo Nn-2{E). As we know that 
yi is in a compact region of i?^, the integration of hi must also be in a compact region. This 
implies that w{g) 7^ only when Xn-2 ^ H'^-2{E) and in a region compact modulo Nn-2{E). 
By the boundedness of Zn-2 as shown in the equation (7.4), we complete the proof of the part 
(1). 

To show the part (2), we keep track of the computation above. Since we are now assuming 
that g e Hn-i{E), we have yi e and hence we may substitute bi biyi. Then for 
^n-i(^i('^n-2) 1)) to be nonzero, we must have bi s 1 + vo^Op. Note that 0„_2 and ^n-2 are 
invariant under multiplication by 1 + w'^Of- We see that w{g) is given by the product of 

rin{yr') \ ct)'^_i{bi{cn-2A))dcn-2r]{hr')d%i, 

JMn-lAF) 



n-2 



\Xn-2\E(pn-2{"e*n_2TXn-2) 



and 



Lpn-2{x^\Un-2an-2)llJr{'^n-2)\an-2\ ^r]n{an-2)dUn-2dan-2- 

J N„-2{F)\H„-2iF) JN„-2{E) 



I Nn-2{F)\H„-2{F) JNn-2{E) 

When y = yi- diag{xn~2, 1), a:n-2 ^ -Bn-2 -(-^), we have 

Sn-l{y) = Sn-2{Xn-2) det{Xn-2) ■ 

Note that 

Now we may repeat this process to complete the proof. □ 

For admissible (fn-i (8) (f)n-i ^ ^c"(^n(^) x Af„,,i(E')), the funciton W,^„_i,</,„_i e W lies in 
^^{Nn-i{E)\Hn-i{E),ipE) and therefore determines a unique element denoted by VF<^„_-^^(^,^_-^ 
in W characterized by 

(7.5) (01)= ^^"-.^"-(^)' 9 e H,,_i{E). 
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Smoothing local Flicker— Rallis period /3„. Let n„ be an irreducible unitary generic 
representation of Hn{E). We now consider the local period /3„. We let W = W(n„,?/^^) be 
the Whittaker model of n„ with respect to the complex conjugate of i/je for later convenience. 
As earlier we have endowed it with a non-degenerate positive definite invariant Hermitian 
structure: 

(W, W) = ^{W, W')= \ W ( ^ )W' ( ^ ]dg. 

We also consider its Kirillov model denoted by /C = /C(n„, iPe)^ which is a certain subspace of 
smooth functions 'Tif'^{Nn-i{E)\Hn-i{E),tlj^). Moreover, the Kirillov model always contains 
the subspace '^^{Nn-i{E)\Hn-i{E),ipE) of smooth compactly-supported functions. 

Let W* be the (conjugate) algebraic dual space of W and 1-i be the Hilbert space underlying 
the unitary representation n„. Then W is the space of smooth vectors in % and we have 
inclusions: 

The Hermitian pairing on W x W extends to "H x 7{ and W x W*. We still denote by n„ the 
representation of Hn{E) on W* so for any W s yV,W' s W*: 

(n^{g)W,W') = (WMg-')W'). 

Then the local Flicker-Rallis period /3„ is an element in W* defined by the convergent integral 

(7.6) /3n{W) = f w( '"-^^ , ] Uh)dh. 

■JNn-i{F)\Hn-i{F) \ ^ J 

To ease notation, we write this as 

J N„-i{F)\H„-iiF) 

We also write this as 

Pn{W)=(W,/3r,). 

It is (i/ri,(F), r7„)-invariant ([9]): 

/3„eHomH„(F)(W,C,J. 
We would like to smoothen the local period /3„ by applying some sort of "moUifier" . 

Proposition 7.2. Assume that the pair ipn-i e {Hn{E)) , (f>n-i e 'rf^ {Mn,i{E)) is m- 
admissible for m > 0. Let W^^_-^^^^_-^ e W be the element determined by 7.5. Then for every 
W E W(Iln,ipE) ! '^^ have 

<n„((^:_i)n„(C_i)w^,/3„> = {w,w^^_,,^^_,). 

In other words, the linear functional n„((^.„_i)n„(y9„_i)/3„, a priori only in W*, is indeed a 
smooth vector and is represented by VF^„_i,</,„_i ^ W(n„, ■?/'£;) . 
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Proof. The LHS is given by 

^n{K^l)W{en-lhgn-lWn-l{9n-l)dgn-lVn{h)dh. 

Substitute gn-i ^ /i"^e~\^„_i, 

Un{(l)l_i)W{gn-l)^n-l{9n-l(^n-lh)dgn-lVn{h)dh. 

JNn-iiF)\H„-i{F) Jh„-i{E) 

Since W{ugn-i) = ipEi'^)^i9n-i) for u e Nn-i{E), we may rewrite the integral as 

^n{K-iW{gn-i) I Tp^_^{g-^^u-^en-ih)ipE{u)du\rin{h)dhdgn-i, 

where the outer integral is over 

h E iV„_i(F)\//„_i(F), e iV„_i(^)\i/„_i(^). 

Now we also note that 



JMn-1,1 



w 



gn-i 



ln-1 

1 



= W{gn^i) (p^^_^{-u)ipE{el_^gn-iu)du 

= W (gn-i) (Pn-i{-el_^9n-i)- 
This completes the proof. □ 

Smoothing local Rankin-Selberg period A. Now let 11 = n„ (x) n„_|_i be an irreducible 
unitary generic representation of G'{F) = Hn{E) x Hn+i{E). We now need another com- 
pactness lemma. For an integer m' > 0, let ^^?°(M„ i (£'))*, be the subspace consisting of 
(pn £ ^c°°(^'i,i(-^)) such that the Fourier transform (p is supported in the domain 

{(xi, E Mn^i{E)\\xi/xn\ < \r^Er',i = l,2,...,n- 1.} 

Lemma 7.3. Let 0„ e ^^°°(M„^i(i?))*,. Let W e W{Iln,ipE) ■^'"c/i that its restriction to 
Hn-i{E) has support compact modulo Nn-i{E). If m' is sufficiently large (depending on the 
vector W ), the map 

H^{E) B g ^ Ue:g)W{g) 
defines an element in (Nn{E)\Hn{E) , ijj ^) . 
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Proof. Clearly (j)n{e^g) is A'„(i?)-invariant and smooth hence the product defines an element 
in {Nn{E)\Hn{E) , -0^). It remains to show that the product has support compact modulo 
Nn{E). By the support condition of (pn{e*g), we may assume that the lower right entry of g 

is non-zero. Therefore we may write g = xu ^"^^ 1 ) where h s Hn-i{E),u e 

Nn,{E),z E Mi^n-i{E) and x e E^ . By the assumption on 0„, we may assume that \\z\\ < 
I^eI™ (otherwise the product vanishes). We may choose m' sufficiently large so that W is 
invariant under right multiplication by 1 + vu^ Mn^niOE) (such ml exists since W e W). We 
now have 



X 



4>n{elg)W{g) = i;E{u)ci)n{x{zA))W 

= 'il:E{u)uJu„{x)(j)n{x{z, 1))W 



h 



h 



1 



where uu„ is the central character of n„. Since the last factor has support compact modulo 
Nn-i{E), and x lies in a compact region, the proof is complete. □ 

Since the Kirillow model IC(Iln+i,ipE) contains {Nn{E)\Hn{E) , ip ^) as a subspace, 
we may view the product 0„,(e*5')H^((7) as an element in /C(n„,+i, ?/^^). This also deter- 
mines uniquely an element denoted by W^^ in the Whittaker model W{Iln+i,ipE)- other 
words, to each W whose restriction to Hn-i{E) lies in {Nn-i{E)\Hn-i{E) , ip ^) and a 
(f)n E ^^°°(M„,i(£;))J„, for m' sufficiently large, we associate an element W^f,^ e yV{Iln+i,ipE) 
characterized by 



(7.7) 



Ue:g)Wig), gEH^{E). 



Its complex conjugate defines an element in >V(n„+i, -^s)- 

Now recall that the local Rankin-Selberg period is defined by the convergent integral, cf. 
(3.13): 



X{W®W') 



W{g)W' 



Nn{E)\Hn{E) 



dg, 



where W e W(n„,^^),iy' e W(n„+i, z/;^^). For 0„ e ^^'*(M„,i(^)), we have an action on 
W(n„+i,^E) by: 



n„+i(0„)iy'(^7) = f w 



ln-1 U 



{u)du, g E Hn+i{E). 



Proposition 7.4. Assume that W e W(n„,'0^) whose restriction to Hn-i{E) lies in 

^-(iV„_i(i?)\/7„_i(E),V^^) 
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and that (f)n e ^^°°(M„ for m! sufficiently large. Then for every W e W(n„+i, ipE) we 

have 



Proof. For g e Hn{E), we have 



w 



A^n-l,l(£;) 



J 



'ipE{elgu)(f){u)du 



liE) 

=Me:9)W' 
Return to the local Rankin-Selberg period: 



Jn„{e)\h„{e) 

= f W{g)M<9)W' 
Jn„(e)\h„{e) 



dg 
dg 



□ 



This completes the proof. 

8 Local character expansion in the general linear case 

In this section we prove a "limit" formula for the (local) spherical character in the general 
linear side. We will choose a subspace of test functions supported in a small neighborhood 
of the origin of the symmetric space Sn+i- Then we may treat them as functions on the "Lie 
algebra" s„+i of S'„+i, the tangent space at origin. The key property for these functions is 
that their Fourier transform have vanishing unipotent orbital integrals except for the regular 
unipotent element. The intermediate steps are messy and somehow ugly; but the final out 
come seems to be miraculously neat. 

A "limit" formula for the spherical character /n(/)- We now combine the two propo- 
sitions above to obtain a formula of /n(/) for a special class of test functions /. Now let 
n = n„ (X) n„+i where the central character of both n„ and n„+i are trivial on F^. 

Consider e ^^°°(i7„(F)) and f^+i e (//„+i(F)). Let 0^ e ^^°°(M„,i(F)). We consider 
a perturbation of /„+i by (f)n 



ftUg) 



fn+l 


f In u\ 


A J* 


hi„,iiE) 





(/){u)du, g^Hn+i{E). 
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This is the same as 0„ o — the convolution introduced in §7 — where we view M„ 

as a subgroup of Hn+i{E). Similarly, for tpn-i e ^c°°(-ff„-i(-E'))), 0„-i e 'if^Mn-i^i{E)), we 

define a perturbation of /„: for g e Hn{E), 



Jm„ue) Jh„-i{ 



12 



Equivalently 



X ^ \ I 1 —u 
1 



(Pn-l{x)(t)n-l{u)dxdu. 



We will consider functions of the following form: 

Definition 8.1. Fzx 11. Let (m, m', r) be positive integers with r > m' > m > 0. We say that 
f = fvn-i,<t>n-i (g) y^^^ (^^^ r)-admissible or admissible for 11 if it satisfies the following: 

• The function (pn-i(E)4'n-i is m-admissible. Hence it determines an element W^^__^-^ e 
%^{Nn^AHn^i{E),il^E) (cf Lemma 7.1) and e >V(n„,V'i^). 

• The function 0„ e '^^°°(M„^i(i?))*, /or sufficiently large m' (depending on n„, (/'n-i ® 

fl'T'C? hence on the integer m). More precisely, m' is large enough such that Lemma 
7.3 holds for W = W^^_^^^^_^. Let VF<^„_i,0„_i,<^„ e W(n„+i, -0^) be the function charac- 
terized by the equation (7.7) for the choice W = W^^_^^^^_^. 

• The function fn (fn+i, resp.) is a multiple of the characteristic function of Mn,n{0 e) 
(of I + w^'Mn+i^n+iiOE) , Tcsp.). Wc normaUzc fnjn+i by 



fn{g)dg = fn+i{g)dg 

JHniE) JHn+l{E) 



We require that r is sufficiently large (depending on U, Lpn-i, 4>n-i, (pn o,nd hence on 
m, m' ) so that 

n„(/n)W(^„_i,0„_i = w^</p„_i,0„_i 

and 

Remark 8. It will be useful later to note that if / = /^"-i'"^"-! (x) /^^-^ is (m, m', r)-admissible, 
then /i t/ is (m)-admissible. 

Proposition 8.2. Fix 11 and assume that f = f^n-i,4>n-i (gi f^"-^ is {m,m' ,r)- admissible for 
n. Then we have 
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Proof. First we have 

/n(/n(X)/n+l) = Yj A(n„(/„)IV ® n„+i(/„,+i)W^')A^ ■ Pn+l{W) 
W,W' 

= X{W®nr,+l{fn+l)W')/3n{U^{f*)W)-/3r,+l{W'), 



WW' 



where the sum of W {W, resp.) runs over an orthonormal basis of n„ (n„+i, resp.). 

For simphcity, we write for 4>n-i and Lp for ipn~i- Now we replace by f^'"^ = / o o 
Note that 

n„((/„^'^)*) = n„(y.*)n„(0*)n„(r) 

By Prop. 7.2, we have for all W s >V(n„,^£;) 

For any Wq e W(n„, 'i/'^;), we have an orthonormal expansion 

Wo = J](Wo,W)W, 
w 

where the sum of W runs over an orthonormal basis of n„. Hence we may fold the sum over 
W to obtain 

W 

Now we further replace by /^"^ = (j)n°fn+i and assume that / = fn'^®ft+i is admissible. 
By the admissibility, we have 

and 

By Prop. 7.4 and fn{9)dg = 1, we have for all W e W(n„+i, -^s): 

Then we may fold the sum over W to obtain 



This completes the proof. □ 
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We need to simplify the formula. Define for a e y4„(F): 
(8.2) Sn{a) := det{Ad{a) : n„))/ det(v4(i(a) : n„_i). 

To simplify the exposition from now on we redefine f^'^i as 0^ o fn+i- 

Proposition 8.3. Fix 11 and assume that f = f^n-i,<f>n-i (g) f^^^ is (m,m, ,r)-admissible for 
n. Use notations as in Lemma 7.1. Then we have 



/n(/)=c^n„(r)|r|*' f f <l>'{b)db] 



(pn-i{~yi{zn-i-i, l)T)\6n{y')\ r]{y')d*y'dz 



where the integral of d*y'dz is over y' e An{F), z e Nn~{F): 

( yovm-'-yn-i \ 



yovi 



^An{F), 



Vo J 



and 



n-l 



h 



Remark 9. Note that |r 
distribution 

I |-(rfn+(in+l)/2 ■ 



\dn 

\E 



^ j E iV„,_(F), z, E Mi,,{F). 
det{Ad{en-i) : Nn-i{E))\. If we consider the normalized 



|, lg 7n(/) = Idet(Men-i) : iV„-i(F))|/'| det(Men) : Nn{E))\^"'luU'), 

then it is independent of the choice of the generator r (up to a sign). 
Proof. By (7.7), we have (cf. (7.6), and note to replace (j)n by 0^) 

/3n+i(M/^¥',0,0u) = W^^^{enh)(})n{-elenh)r]n+i{h)dh. 

JNn{F)\Hn(F) 

Let h = y'z' as in the Lemma and 

y = yo(^^ ^ ) EAn{F),yEAn-i{F), 



1-1/2. 



and 
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Then we may replace the integral on the quotient Nn{F)\Hn{F) by the integral over y', z' for 
the measure: 



n— 1 n—1 

1- 



i=0 i=l 



Since (5n(y') = <^n-i(2/) det(?/) = we may also write this as a product 

where d*y = IULo^'^*^* ^'^'^ '^'^ ^ 0^=1 "^-^j- Since the central character of n„ is trivial on 
F^, we have 



very small so that j ^ ^ ] acts trivially on W^p^^. This allows us to write the integral as 



By the admissibility, the value e*e„/i) = |/o(z„_i, l)r) is non-zero only if l^^n-il is 
the product of 

j0n(-2/o(^n-i, \)T)r]r^+\{yl)d* y^dzn-i 

and 

^n-iyZ \ |e /^.M-lL.I-l 



Ja„_i(F) JAr„_i,_(F) V ' 

where we have used the equality 



By definition W^^fj, ^ ^" ^ ^ = W^p^^{en-iyz) (cf. (7.5)), we may apply the formula of 



the latter by Lemma 7.1: 



( f 0'(5)rf6] ?7„(z/)|5n-l(l/)|Fn<^i(?/*(2;n-i-l,l)T). 

Finally we note rjnijn+i = fj and ^n+i(z/o) = viVo^) = viUo)- This completes the proof. 



□ 



Truncated local expansion of the spherical character /n. We are now ready to deduce 
a truncated local expansion of In around the origin. As we have alluded in the Introduction, 
this expansion is the relative version of Harish-Chandra's local expansion for characters of 
admissible representations of p-adic reductive groups in terms of Fourier transform of nilpo- 
tent orbital integrals. Here we obtain a truncated expansion that only involves the regular 
unipotent element. 
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First we associate to / e 'io^{Hn{E) x Hn+i{E)) with small support around 1 a function 
on the Lie algebra s with small support around 0. To / = /„, ® /n+i ^ ^^{Hn{E) x Hn+i{E)) 
we associate a function / e ^^{Hn+i{E)) by 



= f fn{h)fn+i{hg)dh. 
Jh„(e) 



In other words, we have f = ° fn+i- Moreover, we also have 

The fiber integral defines a map denoted by u : ^ ^^°°(5n+i(F)): 



:= f{gh)dh, iy{g) = X, 

if n is even, and 

^{f)ix) '■= f{9h)r]'{gh)dh, y{g) = x, 

when n is odd (then this depends on the auxiliary character rj'). The Cayley map is a local 
isomorphism near a neighborhood of e s 

1 + X 1 + X 



1 + X 1-X 
and its inverse is given by 

—\(\ \ — X 

c ix) = . 

In particular, for a function $ e ^^{Sn) (0 e ^^'^(S'n), resp.) with support in a small 
neighborhood of 1 e S'„+i (0 e s, resp.), we may consider it as a function on s (Sn+i, resp.) 
denoted by c^^($) (c(0)), resp.). We also have a morphism: 

= t-n+l '■ S Hn+l,E 

such that, wherever they are all defined, we have u o l = c : 

Hn+l{E) 



n+l 
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We associate to / e ^^(Hn(E) x Hn+i{E)) a function on s denoted by 
.3) 



h{X) := f m+X)h)dh, Xes. 
Jh„{f) 



Then we have when det(l — X) 7^ 

c-\v{mX) = f^{X), Xgb. 

From now on all of test functions will be supported in small neighbourhoods of the obvious 
distinguished points of Hn{E) x Hn+i{E), Sn+i{F) and s{F). In particular, we have /t, e 

We then consider s(F) as a subspace of Mn+i,n+i{E) . On Mn+i,n+i{E) we have a bihnear 
paring {X, Y} := ti^XY), under which the decomposition Mn+i^n+i{E) = M„+i^„+i(F)0s(F) 
is orthogonal. We then define the Fourier transform on s(-F) with respect to the restriction 
of the above pairing: 

f^iX):= ( f^{Y)^{tT{XY))dY, 



where the measure is then the self-dual measure for the Fourier transform. We will consider 
the orbital integral of the regular unipotent element 



/ ... \ 

1 ... 

... 1 

V ... 1 y 



G5(F). 



Theorem 8.4. Let II be a unitary generic irreducible admissible representation. Then for 
sufficiently admissible f e ^^{Hn{E) x Hn+i{E)), we have 

/n(/) = |r|g"^'^"-)/V(r)-/i^_U), 
where Un = Wn„ is the central character o/n„,. 

Remark 10. Note that is independent of the choice of r (up to a sign). 

The proof will occupy the rest of this section. 

Determine /[,. We consider admissible functions of the form j¥'"-i'<^"-i (g) /^^^ — note the 
change to (p^ to ease later exposition. Let P — not to be confused with the mirabolic P„ — be 
the subgroup P of Hn+i{E) consisting of elements 



1 * 
\ 1 / 



sHn+i{E) c M„+i,„+i(E). 
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The triple (v9„_i, (^n-i, 0n) then defines a function on \1/ by 



iPn-l{x)(pn-liu)(pniu') . 




/ * 



* 



* 



* 



* 



* 



* 
\ / 



The map p i-^ 1 + p from p to P defines a local homeomorphism from tz7p((9^) to its image. 
We decompose p = p+ ©p^. Then the function \1/ is supported in the subgroup of P{E): 



Note that this is a unimodular subgroup. So either the left or right invariant measure on P{E) 
restricts to a Haar measure on it. Let 0„_i, 0„) be (m, m')-admissible. We may write 

(^j = 0+ (X) 0~ (i = n — l,n) according to the decomposition Mj_i(£') = Mj_i(F) Mi^i{E~) 
and similarly for (fn-i viewed as a function on M„_i„_i(£'). Then we define and \E'^ as 
functions on 1 + p^iOp) and p~ respectively: 



We may even assume that is a multiple of the characteristic function of the subset 
of p"'' consisting of matrices [xij) where Xij = 5ij{modw^) when j ^ n and Xi^n+i = 
Si^n+iip't'Odw^ ). Then it is elementary to check that ^ = '^^ (g) and 



whenever p+ e supp{^^). 

Lemma 8.5. Fix 11 and assume that = y¥'"-i'<^"-i g) admissible for 11. T/ien ti;e 



P(u70) p(wOe) = 1 + ^{wOe). 



? = +. 



(8.4) 




p 



where is the function on s associated to fn ® fn+i o^nd 




is a constant. 
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Proof. We consider the case when n is even; the case n even is similar and only require us to 
change notations several places. By definition we have 

Since /„ a multiple of li+ro^M„,„(c>£;) for sufficiently large r and '\f{g)dg = 1, we may assume 
that o f!^l^ is the same as ftli- 

r = ° C-i ° /nil = ° C-i ° C ° /n+i- 

Explicitly, this is 

Let's denote the RHS by fn+iid)- By the admissibility, the function is of the form \E'+0vl/^ 
where s ^^{p-). By our choice, /„+i is a multiple of li+^'^'Mn+i „+i(c»e) sufficiently 
large r'. In particular, is conjugate invariant under 1 + zuEMn+i,n+i{OE)- 

fn+l{hgh~^) = fn+l{g), hel + WEMn+l,n+l{OE)- 

By the support condition, f^{X) vanishes unless X e s(Ci?). We thus assume that X e 5{Of)- 
Then by definition and the support condition of 

f^{x) = \ \ + x)h)dpdh 

JH„+i{F) JP{E) 

= ( f f ^{p+{i + p.))U+i{p+ii+p-){i + x)h)dp+dp_dh. 

Jh„+i{f) Jp{f) Jp- 
By the invar iance of under the support of we have 

f f f ^+{p+)^-{p^)Ui{{^+P-){l+X)hp+)dp+dp^dh 

JH„+i{F) JP(F) Jp- 

= \ f f ^^{p+)^-{p-)fn+l{{l+P-){l+X)h)dp+dp^dh 

Jh„+i{F) Jp(f) Jp- 
=c(v[/+) f f ^-{p^)U+i{{l+P-){l + X)h)dp_dh. 

JH„+i{F) Jp- 

Note that 

(1 + + X) = (1 + p-X){l + (1 + p^X)-\p^ + X)) 

and (1 +P-X) G 1 + cuM„+i_„+i(Cir) c Hn+i{F). We have 

f f ^-{p-)fn+i{{l+P-){l + X)h)dp_dh 

JH„+i(F) Jp- 

= f f *-(p-)/n+i((l + (l+p_X)-i(p_ + X))/i)rfp_c//i. 
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Compared to the definition of we obtain 

ff{X) =c(v[/+) f v[/-(p_)/^((l + p_X)-\p_ + X))dp^. 
Jp- 

Finally note tliat f\^{X) is a multiple of lro''s(C'F) fo'^ some large integer r. It follows that 
/n+i,^(-^) = fn+i,\\{hX) for any /i e 1 + wMniOp)- Since p. e supp{^!~') c p"(ti7C) and 
X e s{0), we therefore obtain 

/^'^(X) = c(vl>+) [ vl>-(p_)/„^,^^(p_ + X)dp^. 
Jp- 

□ 

Local constancy of the orbital integral. To compare with the unitary case in the next 

section, we need to understand the orbital integral of /*, at least around zero. We show that 
the orbital interval is locally constant around zero and the constant is essentially given by the 
regular unipotent orbital integral. The result will be used in the next section. 

First we study the support of especially its restriction to the relative nilpotent cone 

^^. 

Lemma 8.6. Let f be a sufficiently admissible function. Decompose Sn+i = Sn®s^ and we 
may write as a tensor product of f\ ^ e ^^°°(s„) and some function on s^. Then we have 

(i) The f^ ^ on Sn is supported in s.„^_. 

(a) Let A = (Aij) E Sn be in the support of f\ ^ . Assume further that the absolute values 
|tr A* v4| (2 = 1, 2, n) are bounded by a constant c. Then the norm of the last column 
Vn = (Am)"=i must be bounded (i.e., \\vn\\ is bounded) by some constant depending only 
on c, n, and the norms of the first n ~ 1 columns 1 ^ z < n — 1. 

(Hi) The intersection of the support of f\^ and the relative nilpotent cone Af ^ s is in the 
Hn- orbit of 

Proof. Suppose that / is (m, m', r)-admissible of the form /*. By the admissibility, there 
exists some integer m > such that 

• if y4 is in the support of /Jg, then A = (74jj)"j^]^ satisfies 

(8.5) < K i < n, j ^ ? - 1, 1< i < n, 

• /[,1s is invariant under left and right multiplication of 1 + Mn-i{0 p) (considered as 
subgroup of Hn-i{F), hence of Hn{F)). 
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A 



{modwOp)- 



• \Aii^i\ are fixed non-zero constants depending only on m. 

To show the first part, we may divide the z-th row by and rename the resulted 

matrix by A which lies in It suffices to show A e By the inequality above, we have 

^ * * * * \ 
1 * * * 
1 * * 
\ 1 * J 

Then it is clear that A e f)„^_. This completes the proof of the first part. 

To show the second part, by the invariance of under 1 + cc7™M„_i(Ci?), for A in the 

support of /t]^^, there exists /i e 1 + ■w'^Mn-i^Op) such that hAh^^ remains in the support 
and is of the form 

{ * * * * ^ 

ai * * * 
a„_2 * * 
V an-i * / 

In particular, the absolute value |aj|'s are non-zero fixed constants. Then we may prove that 
the entries of the last column of hAh^^ is a polynomial of the entries of the first n — 1 columns 
of HAh^^ possibly inverting Oi's and the coefficients of characteristic polynomial of A7 This 
shows the last column of HAh^^ satisfies the the assertion of (u). Since /i is in a compact 
group 1 + tu"^ Mn-i{0 f) , the assertion for A follows too. 

^0 



We now prove the third part. Let Xq = e s be in the support and 

unipotent. We need to show Xq e s„+i The i/„-unipotence implies that A^ = 0. Since 
obviously Aq must also be in the support oi f^^ ^ , by the assertion of part (ii) we may assume 
the last column has norm bounded by a constant depending only on n, m, the fields E/F. By 
the condition on the support of we have \Xn+ij/Xn+i^n\ ^ |^"*'| for some large m'. When 
the support of fn+i is small enough (equivalently the integer r is large enough), we clearly 
have for /i E 1 + ro'"M„,„(CF): 

f^ihXoh-') = %{X,). 

Now we may choose a suitable h e l + zu'^Mn^niOp) such that the last column of Xi = hX^h^^ 
is of the form 

(0,0, ...,0,a„, *). 

Ai 

Then it suffices to show that Xi ' 



/[,, the matrix Ai is in the support of /[, 
Ai E s„_. This completes the proof. 



^ 5n+i-- Since Xi remains in the support of 
Frm the first part of the lemma, it follows that 



^To see this, let the last column of hAh~^ be (^ni)i=i,...,n- Then we may go by backward induction on i: 
if z = 71, clearly the first n — 1 columns and the trA can generate A„„; in general, we may show that Ani is a 
polynomial of the first n — 1 columns possibly inverting Oj's and tr a-' A for j = 1, n — i + 1. 
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□ 



We now are ready to prove the local constancy property. This can be viewed as a truncated 
form of Shalika germ expansion (around zero) of the orbital integral of /(, for admissible /. 

Proposition 8.7. // / is sufficiently admissible, then there is a small neighborhood 2" of 
X = such that ri{A^{X))0{X, f\^ is a constant for regular semisimple X. And the constant 
is given by the regular unipotent orbital integral: 

O(X,7^)=77(A_(X/r)K_(70. 

Moreover, for any fixed fixed compact neighborhood of X = ins, there exists an admissible 
f such that the above equality holds for all regular semisimple X in and the RHS does not 
vanish. 

Proof. We first assume that /[, is supported in s„+i Then it follows from Prop. 6.9 (or 
instead, its transpose) that 

r^{^^{X/r))0{XJ) = ^^^_U) 

is a constant in a small neighborhood of zero. In general, we may choose a function /o g "lo^is) 
supported in Sn+i- with the same regular unipotent orbital integral as /[,: 

/^5-(/o) = i^i-{h)- 

By the previous lemma, the functions /o and /[, have the same image in S{M)Hn,'q- Equiv- 
alently, for any (if„, 77)-invariant distribution T supported in A/", T(/o — /^) = 0. We claim 
that 0(X, /o — /^) is zero when X is in a small neighborhood of e s. Indeed, this follows 
from a variant of Bernstein-Zelvinsky localization principle (cf., [28, p. 513, Lemma 27.1]): for 
TT : s„+i ^ ^ we have 

lim^,"(7r-i(^))H„,, - K"(vr-^(0))H„,„ 

where U runs over the set of open neighborhoods of in Since 7V = 7r-i(0) and the image 
of (/o — /[]) in ^c°°('^^^(0))^^n,»? is zero, there exists some neighborhood U oi Q such that the 
image of (/o — /^) in {n^^ {U)) Hr,,r) is zero. This completes the proof of the first part. 
To show the second part, we start with any (mo, mg, ro)-admissible 

f' = f = f^-'''^-'®ftli 

with the following property: 
(1) The regular unipotent orbital integral fi^_{f^) 0. 
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(2) The function (pn e ^(Mi^„(E'))*^ is chosen such that 

where 0^ e 'tf^i^E) is a suitable function which we will keep fixed. 

We then assume that the local constancy holds for a non-empty open neighborhood 3fo of 
0. We claim that if we increase uiq sufficiently and then let Tq he sufficiently large, we can 
maintain the local constancy on Indeed, since the orbital integral depends only on 
orbits, it suffices to consider orbital integral of regular semisimple elements of the form V(a, 6), 
the transpose of a{a,b) (cf. §6 for definition) for (a, 6) in the image of i2o in We may 
replace the integral over Hn{F) by the integral over an open dense subset 

h = ku, u = ( ^"""^ ) ,Vn-i e Mi,„_i(F) 



* * 



and k is in P„ consisting of g Hn- By definition we have 

0{XJ'^=[ [ f(^{u-^k-^Xku)7]{h)dkdu 

where dk is the left invariant Haar measure on P„. For X =* a{a,b), the last row of h^^Xh 
is of the form a„(w„_i, 1) where a„ e is the last entry of k. By the assumption on (pn 
(or rather, we see that the integrand is non-zero only if ||t'n-ill ^ leal™"". Moreover, our 
X =* cr(a, b) lies in a bounded domain If f!^{u~^k~^Xku) 7^ 0, then u^^k^^Xku lies in the 

support of /^', By part (n) of Lemma 8.6, the last row of u^^k^^Xku must be in a bounded 

/ A ^ 

domain with radius depending only on mo, but not on nor Tq. Write X = I 

where A e Mn{F). It follows that when mj, becomes sufficiently large, for X =* a{a, b) e 3fQ, 
we have f^{u^^k^^Xku) = f^{k^^Ak) (also cf. the proof of Lemma 8.8). We thus obtain 

0{XJ{)= I I l^WA/.„_.(o,)K-i)rf^^n-i) I J{{k-^Xk)r^{k)dk. 

As the first factor is a constant and the second factor does not see mp, the claim now follows. 

We return to the proof of the second part. By the compactness of we may assume 
that tS" c ^ for some t e with sufficiently small norm \t\. Now we may replace (^„, by 
(^„,^( which is of the same form but we change mj, to a sufficiently large m'^ and change 

€.(^n)=0L(x.A). 
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We also choose fn-i,t, 4>n~i,t such that = {(Pn-i, 4>n~i, 4>n) has the property 

Note that {(pn-i, 4>n-i) remains mo-admissible (but with A replaced by another lattice tA, cf. 
§7 for definition). Now we let m'l > nii be sufficiently large to fulfill the second condition in 
the definition of admissible function (cf. §8, Definition 8.1). Then with sufficiently large ri 
such that = 1 + w'^^Mi{OE), i = n,n + 1, the function 



ft ~ /? 



is (mo, m']^, 'ri)-admissible (with smaller support than /'). Then we claim that with the new 
choice, the local constancy holds on S'^/t which contains Indeed, we may choose ro,ri 
sufficiently large such that ri = tq + v{t). Then we see that 

n^^{x) = f^{x/t). 

Then their Fourier transforms satisfy 

f^(X) = cJ{{Xt). 
for some constant Q ^ depending only on t. Then clearly 

0{xS,^) = c,0{txJ{). 

The local constancy of 0{X, /^') holds on S'q as shown earlier. This completes the proof of 
the second part of the proposition. □ 

A formula for the regular unipotent orbital integral. We now consider the regular 
unipotent orbital integral of Fourier transform of on s. Recall that the regular unipotent 
orbit is given by 

/ ... \ 
1 ... 
... 1 

V ... 1 ; 



E S. 



Lemma 8.8. Fix U and assume that = f^ri-i,<t>n-i g) f^^^ is admissible for 11. Then we 
have 



n-l 



i=l 



li^Sf?) = c{^^)\SnA^n)\-'^'Yl(p--{0) Ylcl)r{-a,iv,^,A)r)\5n{a)\-'v{a)d*adv, 



where Vi e Mj i(F) and <5„(a) is the same as bef( 



ore. 
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Proof. Without loss of generality we may assume c(\l/"'') = 1. Assume that is {m,m\r)- 
admissible. We denote /' = By definition we have 

/^?-(?)= f 7^{h-^in+i,-h)r^{h)dh= \ f'i~h-%,^,,^hMh)dh. 

When the support of fn+i is small enough, by the previous Lemma 8.5 the function /' on s 
is of the form \E'^ (x) (p^ g) (p^+i corresponding to decomposition 

where is a multiple of Ij^,^^ .(^^^ normalized so that ^ (j)j^{z)dz = 1 or equivalently 0[^(O) = 

1, for i = n,n + 1. The Fourier transform takes the form ® (p'- ® 0„ where \E' is a 
function on the transpose of p^. The way we choose the notation is such that the function 
f? is of the following form: 



4>i 01 


02 




0n 








0n+l 



We also write /* as the tensor product 

where ip^ g ^^°°(Sn). Therefore we may consider the Fourier transform as t/?^ ® 0^ (x) 0^T|_i, 

or as the tensor product of (X)^^/ 0-^ and ^Su^i 4>J ■ Most importantly for our computation, 
the function (/j^ on s is invariant under left and right multiplication by 1 + w™'Mn{Op). Back 
to the orbital integral, we may write h s H'^{F) as 

h = ankv, k e Nn-iHn-i,v = ( """"""^ ],an^F''. 

Then the last row of h^^^^h is of the form 

a„(t;„„i, 1)t. 

Then by the condition on the support of /' (more precisely, of 0„ or 0~ since 0„ e ^^°°(M„^i(£'))|^,), 
we may assume that ll^n-ill < | is small. By the part [ii) of Lemma 8.6, if m' is large 
enough we have 

f{-h'^Cn+i,-h) = ip-{-k-^^n,-k)(l)-{-an{vn-i,l)r)(t)'~+i{0) 
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if the LHS is not zero (i.e., at least h ^^n+i-h lies in the support.) Note that 0^+1(0) = 1 
by our normalization. We now may repeat the process and utilize the invariance of c/?^ under 
1 + w'^ Mn.niO f) ■ To state the outcome we let 



h 



uav, a 



\ 



\ 



an J 



and write v e Nn-{F) as the product of 
n — 1. Note that u^^n-U is of the form 



e iV„_i+i,_(F) c Nn-{F) for 1< z < 



r 



* * 

1 * 

1 

\ 



If V e Nn-i-i-{F), then the conjugation v ^n~uv does not change the last i rows below 

the diagonal of u^^n-iu. We now view 4>i^ a function on the set of upper triangular 
elements and then consider it as a function on s via the natural projection — denoted by 7 
— from 5 to the upper triangular elements. Then the integral ^Oaif^) is equal to 



'n+l 



(^(f^n-i 1 {-{ua) ^^^ua)duY\4'i {-ai{'Vi^i-,l)r)\5n{a)\ ^ri{a)d*adv, 



where u e Nn{F),v e A^„__(F). When r is large enough, the function is a constant in a 
very large neighborhood of 0. By Lemma 6.2, the last column of is a polynomial of 

the first n — 1 columns. By the same lemma, after a suitable substitution we may write the 
integral over u 



X)0l o7((na) ^^-ua)du= (X)0l 07(0 ^ua)du', 





where the integral of u' is over the space of elements of the form 

* * * \ 
1**0 
1*0 

\ 1 y 
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and the measure du' is induced by du. We now make the substitution u' i~> au'a ^ = Ad{a)u' 
(equivalently, u[,j ^ u[ - Y[i<i<^j for 1 ^ z < n — 1): This yields 



o 7(a ^ua)du' 



<lei{Ad{a) : n„_i) J [^0'^rj ° lW)du' 

n 

:det(Ma) : n.-0|r|^^^"^"-^^"-^»/^n'^r(0). 



(Or more exphcitly d,ei{Ad{a) : n„_i) = nr=/ 1*^*1^* Here the factor 

I |(l+2+... + (n-2))/2 1^ / x|l/2 

comes from the difference between the measure on Ft and . Return to fxooifii)- 

nr. n ^ 

|4,E(en)r/'n'^r(0) rf0r(^a,K_i,l)r)|4(a)rS(a)d*adv. 

i^l Ja„(F)7V„,_(F) 

(Or more exphcitly, 5„(a) = det(y4(i(a) : n„))/ det (^c/(a) : n„_i) = nr=^i^ We finally 

note that (/>„^(0) = 1 to complete the proof. 

□ 

Proof of Theorem 8.4. This follows from Prop. 8.3 and Lemma 8.8. We decompose 
ipn-ii4'n-i as in Lemma 7.1 and similarly Note that 

n n— 1 

W (t)n-i{~yi{zn-i-i, i)T)\6n{y')\'^v{y')d*y'dz, 

t=0 

n—1 pfi— 1 

j=0 1=0 
71 ,>n— 1 



fb n nit — _L 



-i(-2/i(2;n-i-i, i)t)|5„(2/')I v{y')d*y'dz. 



and 



™ n~l n—1 p 

<P'ib)db=Ylc^'riO)Yl U:+(foM 



Now it is clear that 

n-l 



:(vi/+) = [] 0+(x,)dx,n <l>\''ik)dk 
1=1 i=l 
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Note that 



I' \E 

Then Theorem 8.4 follows by comparison with Lemma 8.8 



9 Local character expansion in the unitary case 

Three ingredients from [50]. Let F be a non-archimedean local field of characteristic 
zero. We need to recall the main local results of [50]. There are two isomorphism classes of 
Hermitian space Wi,W2 of dimension n. Denote by = U{Wi) the unitary group. We 
let Vi = Wi -L Ee be the orthogonal sum of Wi and a one-dimensional space Ee with norm 
(e, e) = 1. Denote by Uj the Lie algebra of U{Vi). We have a bijection of regular semisimple 
orbits 

H^iF)\siF)rs ^ HwAF)\uiiF)rsUHw,iF)\u2iF)rs 
A regular semisimple X s s matches some Y e ut ii and only if 

l]{A{X/T))=7]{dlSc{W,)), 

where disc(Wi) e F^'/NE'' is the discriminant of Wi. For /' e ''^^{s) and the pair (/i,/2), 
fi E ^^"(uj), we say that /' match (/i, /2), if for all matching regular semisimple X e s,Y e Ui, 
we have 

v'{A4X))0{X,n = 0{YJ.). 

Here rj' is a fixed choice of character E^ with restriction ri'\px = rj. We also say 

/i E ^^°°(ui) and /' E ^c°°(s) match if there exists /2 e ^c°°(u2) such that the pair (/i, /2) and 
/' match. 

Analogous to the general linear case, to a function in a small neighborhood of 1 g G = 
U{W) X U{V), we would like to associate a function on the Lie algebra u of U{V). For 
f = fn® fn+i ^ "rf^iG), we let / be the function on UiV) defined by 

f{g)-= f fn{h)U+i{hg)dh, QEUiV). 
JU{W) 

If / is supported in a small neighborhood of 1 in G, then / is supported in a small 
neighborhood of 1 in UiV). Since the Cayley map c : u ^ UiV) is a local homeomorphism 
around e u, we may denote u = c^^(fi) ^ (l and to / we associate a function denoted by 
= c^^(/) on u supported in uj. To connect the smooth transfer on the groups to the one 
on Lie algebras, we need: 

Lemma 9.1. Let f e ^r°(G'(F)) and fi e "^'fiUiWi) x UiVi)) he matching functions (m the 
sense of %4) with small support around the identity. Then the functions ?7(2")/^' e ^^(s) and 
fi^^ E '^c°°(uj) match. 
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Proof. The support condition insures that the associated functions /^', /j^^ are well-defined. 
Then it remains to show the transfer factor are compatible: 

v'{A4X))0{X,f;^) = n{g)0{g,f), 

where ^{g) = c{X) e Sn+i{F). This follows from the proof of [50, Lemma 3.5]. □ 

In [50, Theorem 2.6] the following result is proved: 
Theorem 9.2. For any f e ^^{u) there exists a matching f e ^c^(s) and conversely. 

Moreover we have [50, Theorem 4.17]: 

Theorem 9.3. // / and f match, then so do e(l/2, 77, -0)"'^^+"')/^/ and f. 

An important ingredient of the proof of both theorems above is a local trace formula on 
Lie algebra ([50, Theorem 4.6]). Now we only need the one in the unitary case. 

Theorem 9.4. For /i, /2 g '^^°°(u), we have 

\ h{X)0{xj2)dX = \0{xJ,)h{X)dX, 
where the integrals are absolutely convergent. 

A hypothesis. We now return to the spherical character in the unitary case. Let n be an 
irreducible admissible representation of G = U{W) x U{V). We use the measure on U{V) 
determined by the self-dual measure on u via the Cayley map. Consider 

Hypothesis (★): there exists a small neighborhood 1] c G of 1 and a function $ e ^^(u) 
such that 

(9.1) $(0) = 1, 

and for all / e ^c°°(fi), 

J.(/)= ( f^{X)OiX,^)dX. 

Ju 

Theorem 9.5. Assume that tt is tempered and Homj^(7r, C) 0. Let ^ be a matrix coefficient 
of n such that 

( ^{h)dh = 1. 

1. The distribution is representable by the orbital integral of ^ 

0{x,^). 

Moreover, it is a bi-H -invariant function that is locally on G. 
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2. Assume further that one of the following two is satisfied 

• The representation vr is supercuspidal. 

• The group H is compact. 

Then the function x t-^ 0{x, $) is locally constant on the regular semisimple locus. 

Proof. When tc is tempered and Hom//(7r,C) 7^ 0, there exists $ such that \^^^{h)dh 7^ 0. 
Up to a scalar multiphcation, we may assume that '\^^{h)dh = 1. Then the first part is 
proved in [21]. For the second part, it is clear if H is compact. If it is supercuspidal, $ has 
compact support and it follows from a general fact proved in [50] that the orbital integral is 
locally constant on the regular semisimple locus. □ 

Proposition 9.6. Assume that Homj:/(7r, C) 7^ 0. // the group H = U{W) is compact or tt is 
supercuspidal, then n verifies Hypothesis (★). 

Proof. Assume that n is supercuspidal. For a small neighborhood of e u we will identify 
it with a neighborhood of 1 e U{V) via the Cayley map. We choose an open and closed 
i/-invariant neighborhood that contains the above chosen neighborhood of 1 e U{V). To 
truncate $, we restrict the function $ to this neighborhood and denote the corresponding 
function on u still by $. It is important to note that we still have 

(9.2) $(0) = 1. 

Moreover the measure on u is transferred to the measure on U{V).Then $ g ^^°°(u) and for 
all function / e ^^{G) with small support around 1, 

Mf)= \ f^{x)Oix,^)dx. 

If H is compact (so that dimW < 2 or E/F = C/R), then there is a non-zero vector 
(f)o e TV fixed by H. Then for all /, we have 

J.(/) = vol(/f) f fig)(7rig)<PoAo)dg 

JG 

for a norm one 0o ^ Set ^{g) = vol(iJ)^^(7r(5')0o5 0o)- Then the same truncation as 

above completes the proof. 

□ 

We say that / is admissible if there is an admissible /' matching /. 

Theorem 9.7. Assume that tt verifies Hypothesis (★). there exists admissible functions f e 
^^(^(F)) and matching function f s ^^°°(G'(F)) such that 

Mf) = iv\r)/eil/2,rj,^)r(-^'y'rj{dtsc{W))r^M0 * 0. 
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Proof. First we apply the local trace formula (Theorem 9.4) 

\ f^{Y)0{Y,^)dY= (0{Y,%MY)dY, 

Ju Ju 

where $ is the inverse of the Fourier transform. Let /' be an admissible function matching 
/. Then by the compatibility between Fourier transform and smooth transfer (Theorem 9.3) 

e(l/2,r/,^)"("+i)/20(y,/^) = r^'{A^{X))OixJ;) 

for matching regular semisimple X and Y. Since $ has compact support, by Prop. 8.7, we 
may choose a sufficiently admissible functions /' such that 

0{YJ^) = (V(r)/6(l/2,r7,V^))"("+^)/277(A(X/r))/i^_(^0 ^ 

is a constant for all regular semisimple Y in the support of $. We know that for regular 
semisimple X, its transpose X* is in the same //„-orbit and X* = hXh^^ for a unique 
he Hn{F) with 

r^{h) = r/(A+(X)/A_(X)) = r7(A(X/T)) = T^{disc{W)). 
Therefore for all regular semisimple Y in the support of $ we have 

0{YJ^) = (r]'(r)/e(l/2,r/,^))"("+i)/2r;(dzsc(W))/X5_(^0 ^ 0. 

We obtain 

Uf)= \o{x,%)^x)dx 

Ju 

= iv'{T)/eil/2,7J,^p)r^-+'y'r]{d^sc{W))fi^_if0 ■ f ^X)dX 

Ju 

= (V(r)/e(l/2,r7,^^))"("+^)/2r;(dz5c(M/))/i^_(^') ■ $(0). 
By Hypothesis (*) we have 

$(0) = 1. 

The theorem now follows. □ 

Completion of the proof of Theorem 4.5. It remains to prove the case when v is 
non-split. Since the ratio -^n„(/^)/<^7r„(/«) is independent of the choice of f^, we may choose 
a suitable admissible function. Then the result follows from Theorem 8.4, Prop. 9.6 and 
Theorem 9.7. 
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Concluding remarks Note that we only deal with vr^ which appears as a local component 
of a global n. But we expect Conjecture 4.3 holds in general (as long as n„ is generic in order 
to define /n„)- 

We conclude with 

Conjecture 9.8. The spherical character In and J^r are representable by a locally function 
and smooth (local constant in the non-archimedean case) on an open subset. 

There should be a more complete analogue of the Harish-Chandra local character expan- 
sion in our relative setting. Moreover, it seems that the spherical characters (if non-zero) 
should contains as much information as the usual characters of the representation. 
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